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PREFACE 


The  research  reported  in  this  memorandum  was  conducted  under  the 
sponsorship  of  the  Advanced  Research  Projects  Agency  (ARPA)  and  is  a 
product  of  a  continuing  study  of  the  management  of  research  and  develop 
ment  (R&D).  It  presents  a  technique  for  analyzing  the  effects  of  alter 
native  policies  for  decisionmaking  and  alternative  forms  of  contracts 
in  system  development.  A  computer  program  for  generating  histories  of 
decisionmaking  in  system  development  is  developed  and  demonstrated. 

The  program  emphasizes  the  information  and  learning  aspects  of  the  de¬ 
velopment  process.  This  program  approach  constitutes  an  alternative 
method  for  gaining  experience  with  various  policies  and  contracts. 

The  program  should  be  of  interest  to  persons  who  are  concerned 
with  the  evaluation  of  policies  for  decisionmaking  and  the  evaluation 
of  alternative  forms  of  contracts.  The  conceptual  framework  should  be 
of  interest  to  all  persons  involved  in  the  management  of  R&D,  especi¬ 
ally  in  the  relation  of  technical  information  processing  to  risk  assess 
ment.  The  technique  may  be  applied  to  military  or  commercial  develop¬ 
ment  projects. 

In  addition  to  the  support  of  The  Rand  Corporation,  this  research 
was  partially  supported  by  the  Office  of  Naval  Research  (Contract  No. 
N00014-67-0111-0010) ,  and,  in  a  different  form,  also  constituted  the 
author's  doctoral  dissertation. 
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SUMMARY 


The  research  in  this  memorandum  presents  a  method  for  analyzing 
engineering  decisionmaking  in  a  system  development  project.  System 
development  is  viewed  as  having  the  following  characteristics: 

o  A  system  is  developed  by  sequentially  executing  actions  that 
buy  information  about  the  parts  of  the  system. 

o  The  ultimate  determination  of  the  success  or  value  of  the  final 
product  depends  on  the  characteristics  of  the  whole,  not  the 
parts . 

o  At  any  time,  the  information  about  the  parts  and  the  whole  is 
uncertain . 

The  method  is  developed  in  such  a  wav  that  it  can  be  used  as  a 
tool  to  evaluate  policies  for  decisionmaking  or  to  evaluate  the  influ¬ 
ence  of  multiple  incentive  contracts.  The  decisionmaking  policies  ex¬ 
amined  are  organization-oriented  rather  than  individual-oriented.  They 
use  system  and  project  characteristics  as  the  criteria  for  decision¬ 
making  instead  of  lower  level  characteristics,  or  personal  goals. 

The  result  is  a  model  in  which  the  component-level  physical  and 
performance  characteristics  and  the  component-level  engineering  activ¬ 
ities  of  a  svstem  development  project  are  explicitly  related  to  the 
ultimate  results  at  the  system  and  total  program  level.  In  this  model 
activities  are  performed  that  yield  information  about  the  phvsical  and 
performance  charac t eris t ics  of  the  parts  of  the  system.  This  informa¬ 
tion  is  uncertain,  and  it  is  expressed  in  the  form  of  probability  dis¬ 
tributions.  Selection  of  subsequent  activities  is  based  on  the  impact 
of  the  component-level  information  at  the  system  and  total  program 
level;  that  is,  in  terms  of  the  impact  on  system  performance,  contract 
payoffs,  delivery  dates,  and  program  costs.  In  this  memorandum,  the 
uncertainty  regarding  the  system  performance  and  contract  payoff  is 
treated  explicitly  in  the  decisionmaking  process.  Delivery  dates  and 
program  costs  are  treated  using  single-valued  forecasts. 

The  research  combines  engineering  and  decisionmaking  analyses  into 
a  simulation  model  of  system  development.  The  simulation  model  has  two 
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parts:  (1)  a  model  of  a  system  development  project;  and  (2)  a  model 

of  the  decision  and- information  processes  in  system  development. 

The  system  development  project  is  characterized  by  a  set  of  goals 
that  are  expressed  as  a  preference  relationship  over  system  and  proj¬ 
ect  characteristics  in  the  form  of  a  multiple  incentive  contract;  a 
number  of  alternative  designs  for  the  components  of  the  system;  a  num¬ 
ber  of  engineering  activities  that  can  be  conducted  on  the  component 
designs  to  improve  the  state  of  knowledge  regarding  the  components; 
and  a  state  of  knowledge  that  has  two  elements:  (1)  the  relationships 
between  the  characteristics  of  the  system,  the  subsystems,  and  the  com¬ 
ponents;  and  (2)  the  knowledge  regarding  the  values  of  the  system,  sub¬ 
system,  and  component  characteristics. 

The  model  of  the  decision  and  information  portion  of  the  system 
development  process  functions  as  follows: 

1.  Actions  (engineering  activities  applied  to  specific  component 
designs)  are  evaluated.  The  details  of  the  evaluation  procedure 
vary  from  one  decisionmaking  policy  to  the  next.  All  policies 
make  some  kind  of  forecast  regarding  the  technical  results 
expected  from  a  component-level  action.  Some  forecasts  ex¬ 
plicitly  account  for  technical  uncertainty  and  some  do  not. 
Propagation  of  error  methods  are  used  to  determine  the  impact 

of  the  component-level  forecast  at  the  system  and  total-program 

*  v 

level . 

2.  Actions  are  selected  sequentially. 

3.  Actions  are  executed. 

4.  Information  is  received  regarding  the  values  of  component 
characteristics.  This  information  is  uncertain,  and  the  amount 
of  uncertainty  is  a  direct  function  of  the  activity  that  gen¬ 
erated  the  information. 

5.  Propagation  of  error  methods  are  used  to  determine  new  knowl¬ 
edge  regarding  values  of  the  subsystem  and  system  character¬ 
istics. 


This  procedure  can  be  described  as  a  dynamic  technical  risk  as¬ 
sessment  that  is  updated  every  time  new  information  is  obtained. 
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These  steps  are  repeated  until  a  terminal  action  ends  a  given  project 
simulation . 

In  the  research  reported  here,  nine  decisionmaking  policies  were 
considered  and  five  were  selected  for  study  using  samples  of  51  simu¬ 
lated  projects.  (Policies  2,  3,  7,  and  9  were  not  examined,  either  be¬ 
cause  of  inherent  difficulties  or  too  great  a  similarity  to  other  pol¬ 
icies.)  The  policies  selected  for  in-depth  study  represent  the  follow¬ 
ing  conditions: 

1.  Policy  1:  Maximize  Expected  Payoff  is  a  close  approximation 
to  the  decision  procedure  used  in  sequential  decision  theory 
in  which  the  value  of  a  terminal  action  is  compared  to  the 
value  of  an  action  that  buys  more  information.  These  calcu¬ 
lations  are  based  on  the  complete  probability  distributions 
for  the  two  cases.  Thus,  it  represents  a  policy  that  would 

be  recommended  by  economic  decision  theorists.  The  payoff  re¬ 
fers  to  the  fee  earned  under  the  multiple  incentive  contract. 
All  other  policies  are  deviations  from  Policy  1. 

2.  Policy  4:  Current  Means  Analysis  differs  from  Policy  1  in 
that  the  evaluation  of  actions  is  based  on  the  means  of  the 
system  performance  probability  distributions,  while  Policy  1 
utilizes  the  entire  distributions.  Hence,  Policy  4  is  intended 
to  show  the  effect  of  using  only  a  single-valued  estimate  of 
the  outcomes  instead  of  all  possible  outcomes. 

3.  Policy  5:  Low-Cost  Actions  First  differs  from  Policy  1  in  re¬ 
quiring  that  all  of  the  actions  that  are  low-cost  and  short- 
time  be  done  first.  There  are  low-cost  actions  in  all  alter¬ 
native  designs,  hence,  Policy  5  shows  the  effect  of  examining 
all  designs  early  in  the  project. 

4.  Policy  6:  Initial  Prior  Means >  Optimistic  differs  from  Pol¬ 
icy  1  in  not  using  all  of  current  technical  information. 

Policy  6  uses  current  information  for  the  characteristics  that 
will  be  measured.  For  the  characteristics  that  will  not  be 
measured,  the  mean  values  from  the  beginning  of  the  project 
are  used.  Hence,  Policy  6  is  intended  to  show  the  effect  of 
only  partial  distribution  of  technical  information  to  decision¬ 
makers. 
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5.  Policy  8:  Probability  of  Technical  Success  differs  from  Pol¬ 
icy  1  in  using  the  probability  of  technical  success  as  the 
criterion  for  selecting  actions.  Policy  1  uses  the  expected 
payoff  that  includes  time  and  cost  considerations  as  well  as 
the  technical  performance. 

These  five  policies  were  examined  using  two  multiple  incentive 
contracts.  The  zero-incentive  points  for  both  contracts  were  the  same 
and  there  was  only  a  small  difference  between  the  two  with  regard  to 
the  relative  weight  placed  on  the  system  performance  characteristics. 

The  major  difference  between  the  contracts  was  the  rate  of  incentive 
for  being  over  or  under  the  zero-incentive  (required)  values  for  the 
sys tenu  performance . 

An  antisubmarine  warfare  patrol  aircraft  was  used  for  the  system 
to  be  developed.  The  technical  performance  characteristics  used  in  the 
incentive  contracts  are  the  gross  take-off  weight  and  a  fuel  economy 
measure.  The  two  contracts  and  five  policies  resulted  in  ten  differ¬ 
ent  development  cases. 

The  results  were  analyzed  using  two  system  performance  character¬ 
istics  (gross  take-off  weijgHt'  and  fuel  economy),  the  program  cost  and 
time,  the  contractor’s  fee,  the  sequences  of  actions,  and  the  frequen¬ 
cies  of  technical  success  and  failure. 

To  demonstrate  the  use  of  the  technique  in  evaluating  decision¬ 
making  policies  and  incentive  contracts,  several  conclusions  were  drawn 
using  the  results  of  the  ten  sample  cases.  These  conclusions  are  con¬ 
ditional  upon  the  special  example  used  in  this  study  and  should  not  be 
taken  as  evidence  regarding  the  relative  merits  of  the  various  poli¬ 
cies  and  contracts  studied  for  other  projects,  technologies,  uncertain¬ 
ties,  and  so  on.  However,  the  technique  has  many  other  uses,  including 
analysis  of  impact  of  multiple  incentive  contracts  on  engineering  deci¬ 
sions  and  technical  outcomes  and  generation  of  probability  distributions 
for  cost  estimates.  Several  of  these  will  be  briefly  discussed  at  the 
end  of  the  memorandum. 
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I .  INTRODUCTION 


PURPOSE 

The  purpose  of  this  memorandum  is  to  present  a  model  of  engineer¬ 
ing  decisionmaking  in  a  system  development  project  and  to  demonstrate 
the  use  of  the  model  as  a  tool  for  evaluating  policies  for  decision¬ 
making  and  multiple  incentive  contracts.  In  this  work,  system  develop¬ 
ment  is  viewed  as  having  the  following  characteristics: 

o  A  system  is  developed  by  sequentially  executing  actions  that 
buy  information  about  the  parts  of  the  system. 

o  The  ultimate  determination  of  the  success  or  value  of  the  final 
product  depends  on  the  characteristics  of  the  whole,  not  the 
part . 

o  At  any  time  during  the  development  process,  the  information 
about  the  parts  and  the  system  is  uncertain. 

The  motivation  for  this  research  stems  from  many  previous  studies 

of  decisionmaking  in  research  and  development  (R&D) .  Policies  for  de- 

* 

cisionmaking  in  R&D  have  been  the  subject  of  many  studies  and  some  of 
the  policies  studied  in  the  present  work  reflect  these  earlier  studies. 

Most  past  studies  have  dealt  with  development  decisionmaking  in 
terms  of  results  at  the  total  program  level,  for  example,  the  system 
performance,  program  cost,  delivery  date,  and  contractor  payoff.  How¬ 
ever,  the  alternative  courses  of  action  examined  in  these  studies  were 
not  "operationally  defined";  that  is,  they  did  not  specify  actions 
that  engineers  in  development  organizations  actually  carry  out,  there¬ 
by  obtaining  items  of  information  regarding  the  system.  Instead,  cour¬ 
ses  of  action  such  as  "work  on  design  A,"  or  Mwork  on  design  B,"  or 
"work  on  designs  C  and  D"  were  the  alternatives  considered.  This  prob¬ 
ably  was  due  to  the  desire  for  actions  that  produced  information  (or 
results)  at  the  same  level  as  the  evaluation.  The  resulting  studies  were 

* 

See  especially,  B.  H.  Klein,  W.  H.  Meckling,  and  E.  G.  Mesthene, 
military  Research  and  Development  Policies ,  The  Rand  Corporation,  R-333, 
December  4,  1958;  and  Thomas  Marschak,  T.  K.  Glennan,  Jr.,  and  Robert 
Summers,  Strategy  for  R&D ,  Springer-Verlag ,  New  York,  1967. 
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useful  to  academicians  but  of  limited  value  to  practicing  development 
decisionmakers . 

One  contribution  of  the  present  research  is  the  combination  of 
the  concept  of  evaluation  at  the  total  program  level  with  alternative 
courses  of  action  defined  at  the  engineering  level.  This  combination 
should  open  the  way  for  more  and  better  research  on  development — 
research  that  will  be  meaningful  to  both  academicians  and  development 
decisionmakers . 

To  demonstrate  the  unified  framework,  a  computer  simulation  model 
of  a  system  development  project  is  constructed.  This  model  has  great 
potential  in  itself.  It  is  a  tool  that  can  be  used  as  a  "laboratory" 
for  testing  alternative  decisionmaking  policies  and  contracts  for 
influencing  decisionmaking.  The  program  can  be  changed  to  reflect  dif¬ 
ferent  engineering  activities,  alternative  designs,  initial  conditions, 
incentive  contracts,  decisionmaking  policies,  and  even  technologies. 
Some  of  these  changes  require  only  new  input  data  and  others  require 
rewriting  parts  of  the  program,  but  the  general  approach  is  the  same 
for  all  cases.  Any  number  of  project  histories  can  be  obtained  for 
each  set  of  conditions.  Thus >  large  samples  can  be  generated  and  sta¬ 
tistical  evaluation  techniques  used.  The  model  is  described  in  terms 
of  two  sets  of  elements  that  have  been  identified  as  "engineering  con¬ 
siderations"  and  "management  considerations." 

ENGINEERING  CONSIDERATIONS 

The  engineering  aspects  of  the  project  constitute  the  inputs  to 
the  model  and  include  the  items  described  below. 

Goals 

The  goals  of  a  system  development  project  are  represented  by  a 
multiple  incentive  contract  over  time,  cost,  and  system  performance. 
Values  for  the  contract  terms  are  read  in  as  input  data. 

Alternatives 

The  alternatives  that  are  available  refer  to  the  various  designs 
and  configurations  for  the  components,  subsystems,  and  the  system. 
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They  are  characterized  by  probability  distributions  over  the  values  of 

the  characteristics  of  each  part  (e.g.,  component,  subsystem,  etc.)* 

These  probability  distributions  bear  an  important  relationship  to  each 

* 

other  as  described  under  the  Hierarchy  of  Uncertainty  heading  below. 

Activities 

The  activities  are  engineering  processes  that  buy  information 
about  the  characteristics  of  the  alternatives.  Activities  may  be 
applied  at  various  levels  (e.g.,  component,  subsystem,  or  system).  An 
activity  is  characterized  by  a  time,  a  cost,  the  characterist ic(s)  mea¬ 
sured,  and  the  precision  of  the  measurement.  Values  for  these  items 
are  read  as  input  data. 

The  State  of Knowledge 

The  state  of  knowledge  refers  to  what  is  known  at  any  given  time 
about  the  system  being  developed.  There  are  two  important  aspects  to 
the  state  of  knowledge. 

o  Technology .  The  equations  that  specify  the  functional  relations 
between  the  physical  and  performance  characteristics  at  all 
levels  (e.g.,  component,  subsystem,  and  system)  form  one  part 
of  the  state  of  knowledge.  This  research  assumes  that  these 
equations  are  known  and  not  subject  to  any  uncertainty.  The 
model  of  system  development  uses  this  body  of  equations  to  es¬ 
tablish  the  Technical  Information  Subsystem  of  the  management 
system. 

o  Hierarchy  of  Uncertainty .  As  specified  above,  each  alternative 
is  represented  by  probability  distributions  over  the  character¬ 
istics  of  the  alternative.  The  distributions  at  adjacent  levels 
are  related  by  the  design  equations.  The  uncertainty  hierarchy 


* 

To  avoid  any  semantic  difficulties,  these  alternatives  should  not 
be  referred  to  as  alternative  courses  of  action.  Actions  constitute 
the  application  of  some  engineering  activity  to  some  alternative.  For 
example,  wind  tunnel  testing  is  an  activity ,  wing  design  number  1  is  an 
alternative  ,  and  doing  a  wind  tunnel  test  of  wing  design  number  1  is  an 
action . 
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at  the  stai.t  of  the  development  project  is  determined  by  apply¬ 
ing  the  Technical  Information  Subsystem  to  probability  distri¬ 
butions  for  the  lowest  level  characteristics.  These  initial, 
low-level  distributions  are  read  in  as  input  data.  The  hier¬ 
archy  is  subject  to  revision  as  information  is  learned  by  carry¬ 
ing  out  activities  cn  alternatives  (executing  actions).  There 
are  as  many  hierarchies  as  there  are  system  designs.  Propaga¬ 
tion  of  error  methods  are  used  to  update  the  appropriate  hierarchy. 

MAN  A CTMJOT  CONSIDERATIONS 

The  management  aspects  of  a  system  development  project  constitute 
a  complete  management  information  and  decision  (command  and  control) 
system.  Two  major  elements  of  the  management  system  are  identified 
b  e  1  ow . 


Te  cl  i  nic  al  I  n  f_o  rm  a  t  i  o  n  Sub  system 

The  Technical  Information  Subsystem  is  the  body  of  equations  that 
specify  the  relationships  between  the  characteristics  at  the  various 
levels.  In  the  present  work  it  is  expressed  by  the  propagation  of  error 
expressions  derived  from  the  set  of  design  equations  (called  the  Tech- 
nologv) .  These  expressions  are  embodied  in  a  hierarchical  set  of  sub¬ 
routines  in  the  program. 

Decis ion mak i_ng  P o  1  i c  ies 

A  decisionmaking  policy  is  a  collection  of  rules  used  to  evaluate 
and  select  actions.  Three  elements  of  a  decisionmaking  policy  are 
identified  as  follows: 

o  Eligibility.  Rule .  The  first  consideration  in  action  selection 
is  a  means  for  determining  which  of  all  possible  actions  can 
and  may  be  considered  for  selection  at  any  given  decision  point. 

k 

A  system  using  the  design  equations  and  a  Monte  Carlo  computer 
routine  is  described  in  F.  S.  Timson,  Measurement  of  Technical  Perform¬ 
ance  in  Weapon  System  Development  Programs:  A  Subjective  Probability 
Approach ,  The  Ra(nd  Corporation,  RM-5207-ARPA,  December  1968. 
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This  includes  both  the  physical  or  logical  feasibility  and 

* 

management  preference. 

o  Forecasts .  The  second  consideration  is  what  information  will 
be  used  to  evaluate  those  actions  that  are  eligible.  This  in¬ 
cludes  the  portion  of  the  current  available  information  that 
will  be  used  for  making  time,  cost,  and  performance  forecasts. 
Time  and  cost  forecasts  do  not  include  uncertainty  in  the  pres- 
ent  work.  Some  performance  forecasts  explicitly  account  for 
technical  uncertainty  and  some  do  not.  Propagation  of  error 
methods  are  used  to  determine  the  impact  of  the  component-level 
forecast  at  the  system  level. ^ 

o  Evaluation  Rule .  The  third  consideration  is  the  criterion  func¬ 
tion  that  will  assign  values  to  the  eligible  actions  using  the 
forecast  information . 

EXECUTION  OF  ENGINEERING  ACTIONS 

The  decisionmaking  portion  of  the  system  development  process  se¬ 
quentially  selects  engineering  actions  that  buy  information  about  the 
characteristics  of  the  alternatives.  When  an  action  is  selected,  the 
model  determines  a  new  state  of  knowledge  for  the  character is t ic (s) 
measured.  The  time  and  cost  of  the  action  are  recorded.  The  Techni¬ 
cal  Information  Subsystem  then  determines  the  new  values  for  the  char¬ 
acteristics  at  higher  levels  x>f  the  appropriate  hierarchy  of  uncertainty. 

Outputs 

The  model  simulates  the  history  of  engineering  activities  in  a 
system  development  project.  The  information  available  at  the  end  of 
a  project  simulation  is  divided  into  the  following  two  categories: 

The  relation  of  eligibility  rules  to  scheduling  of  activities 
will  be  discussed  in  Sec.  III. 

This  procedure  can  be  described  as  a  dynamic  technical  risk  as¬ 
sessment  that  is  updated  every  time  new  information  is  obtained. 

Inclusion  of  uncertainty  regarding  times  and  costs  is  discussed 
in  Sec.  VII. 
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o  The  Final  State  of  the  Development  Process .  The  final  state 
of  the  development  process  includes  (1)  values  for  the  system 
performance  characteristics,  (2)  the  identification  of  the 
system  design  that  was  delivered,  (3)  the  time  of  delivery, 

(4)  the  sum  of  the  costs  of  all  activities  undertaken,  and 

(5)  the  payoff  to  the  contractor. 

o  The  Development  History .  The  program  presently  keeps  track  of 
two  items  in  the  development  history.  (1)  It  lists  the  actions 
taken,  in  sequence.  It  identifies  the  activity  performed  and 
the  design  and  part  of  the  system  worked  on.  (2)  The  system 
performance  probability  distributions  for  all  possible  system 
designs  are  provided  at  each  decision  point. 

The  model  can  be  run  any  number  of  times  to  collect  a  sample  of 
values  for  all  the  output  items. 

APPLICATIONS 

The  model  is  most  useful  for  evaluating  the  impact  of  alternative 
decisionmaking  policies  and  multiple  incentive  contracts  on  the  re¬ 
sults  of  a  system  development  project.  This  application  is  demonstrated 
in  this  memorandum.  Several  other  uses  of  the  model — including  the 
real-world  management  of  system  development,  the  analysis  of  multiple 
incentive  contracts,  and  the  production  of  cost  estimates  that  include 
uncertainty — are  discussed  in  the  last  section. 

OUTLINE  OF  MEMORANDUM 

The  engineering  considerations  and  the  details  for  the  specific 
project  used  in  this  research  are  presented  in  Sec.  II.  A  general  dis¬ 
cussion  of  the  management  considerations  and  execution  is  presented  in 
Sec.  III.  The  decisionmaking  policies  are  described  in  Sec.  IV. 

Summary  data  on  the  results  obtained  under  the  various  policies 
and  contracts  are  presented  in  Sec.  V.  Section  VI  presents  the  analy¬ 
ses  of  the  policies  and  contracts,  and  some  conclusions  drawn  from 
these  analyses  are  suggested  in  Sec.  VII.  Supporting  materials  are 
presented  in*the  appendices. 


II.  THE  ENGINEERING  CONSIDERATIONS  IN 


A  SYSTEM  DEVELOPMENT  PROJECT 


This  section  discusses  the  engineering  considerations  in  a  system 
development  project  and  presents  the  detailed  description  of  these 
items  for  a  hypothetical  development  project.  The  system  development 
project  used  in  this  research  is  for  an  antisubmarine  warfare  (ASW) 
patrol  aircraft.  This  type  of  project  was  chosen  because  the  technol¬ 
ogy  is  well  understood,  sources  were  available  to  help  with  defining 
the  elements  of  the  hypothetical  project,  the  data  involved  are  non¬ 
proprietary  and  unclassified,  and  the  system  performance  characteris¬ 
tics  that  indicate  the  aircraft’s  mission  capability  are  relatively 
easy  to  put  into  analytical  form. 

GOALS 

All  system  development  programs  have  a  purpose.  This  purpose  is 
expressed  in  the  form  of  goals  regarding  the  final  outcome  of  the  pro¬ 
gram.  For  the  hypothetical  development  program  used  in  this  study  the 
following  attributes  of  the  final  state  are  considered:  (1)  two  per¬ 
formance  characteristics  of  the  aircraft,  (2)  the  time  of  delivery, 

(3)  the  cost  of  the  program — exclusive  of  fee,  and  (4)  the  fee  paid  to 
the  contractor. 

System  Performance 

The  system  performance  characteristics  that  are  used  in  the  con¬ 
tract  should  be  good  indicators  of  the  system’s  mission  capability. 

In  addition,  the  characteristics  must  be  measurable  and  the  method  of 
measurement  must  be  agreed  on  and  specified  in  advance.  For  aircraft, 
certain  performance  characteristics  depend  on  atmospheric  conditions. 
Because  these  conditions  vary  from  day  to  day  and  place  to  place,  it 
is  necessary  to  specify  the  conditions  to  be  used  for  performance  eval¬ 
uation  and  the  means  for  correcting  for  deviations  in  actual  conditions 
when  variations  occur. 
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An  ASW  patrol  aircraft  can  perform  several  different  patrol  mis¬ 
sions.  The  aircraft’s  ability  to  carry  out  these  missions  is  enhanced 
by  its  ability  to  cover  large  expanses  of  ocean  and  to  remain  on  sta¬ 
tion  for  long  periods  of  time.  The  best  system  performance  character¬ 
istics  to  indicate  mission  capability  are  thus  range  and  endurance. 
However,  from  the  point  of  view  of  this  study,  it  is  not  possible  to 
express  these  characteristics  in  an  analytical  form  that  can  be  used 
with  the  propagation  of  error  technique. 

Two  substitute  performance  characteristics  are  used  instead.  These 
are  a  measure  of  fuel  economy  and  the  gross  take-off  weight.  The  fuel 
economy  measure  is  the  number  of  nautical  miles  that  the  aircraft  trav¬ 
els  per  pound  of  fuel  consumed  at  a  given  set  of  flight  conditions. 

These  conditions  include  the  aircraft’s  instantaneous  gross  weight,  the 
number  of  engines  operating,  the  speed  of  the  aircraft,  the  atmospheric 
pressure  and  the  temperature.  The  aircraft-related  flight  conditions 
are  chosen  to  correspond  to  one  particular  mission  and  the  atmospheric 
conditions  are  chosen  to  correspond  to  this  mission  on  a  "standard 
day."  For  the  hypothetical  project  used  in  this  study  a  low  altitude 
loiter  patrol  was  chosen.  This  mission  corresponds  to  the  following 
set  of  values:  (1)  aircraft  gross  weight  at  time  of  mission  =  100,000 
pounds,  (2)  number  of  engines  operating  =  2,  (3)  aircraft  speed  =  200 
knots,  (4)  atmospheric  pressure  =  28.33  inches  of  mercury,  and  (5)  tem¬ 
perature  =  513.4  degrees  Rankine.  The  mission  description  is  summarized 
as  follows: 


Item 


Description  or  Value 


Aircraft  type 
Mission 
A1 t itude 
Airspeed 

Number  of  engines 
operating 


4-engine  turboprop  ASW  patrol 
Low-altitude  search 
1500  ft 
200  kn 

2 


A  "standard  day"  is  a  day  on  which  the  atmosphere  exhibits  the 
properties  shown  in  a  Standard  Atmosphere  Table.  Such  a  table  shows 
pressure,  pressure  ratio,  temperature,  density,  specific  weight;  den¬ 
sity  ratio,  the  speed  of  sound,  and  the  kinematic  viscosity  as  a  func¬ 
tion  of  altitude.  See  Court  land  D.  Perkins  and  Robert  E.  Hage ,  Air¬ 
plane  Performance  Stability  and  Control ,  John  Wiley  &  Sons,  Inc.,  New 
York,  1949,  pp .  481-483.  The  "standard  day"  is  used  as  a  basis  for 
almost  all  aircraft  performance  analysis. 
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The  gross  take-off  weight  is  related  to  the  aircraft's  mission 
capability  in  the  following  way:  The  aircraft  is  designed  to  operate 
at  a  given  gross  take-off  weight  that  includes  a  certain  amount  of  fuel. 
If  the  actual  gross  take-off  weight  exceeds  the  design  gross  take-off 
weight,  there  are  two  general  courses  of  action:  (1)  Use  the  existing 
airframe,  but  trade  off  fuel  and  payload,  which  will,  of  course,  de¬ 
grade  mission  capability;  or  (2)  at  the  expense  of  additional  time  and 
money,  develop  another  airframe,  in  the  hope  it  will  provide  the  de¬ 
sired  capability. 

Time 

Incentive  contracts  may  relate  fee  to  a  number  of  different  con¬ 
tract  performance  times  corresponding  to  the  accomplishment  of  certain 
"milestones"  by  the  contractor.  In  this  study,  only  one  time  is  con¬ 
sidered —  the  time  of  delivery  to  the  customer. 

Cost 

The  cost  used  in  this  study  is  the  cost  of  carrying  out  the  con¬ 
tract,  but  excludes  the  fee  earned.  It  is  the  sum  of  material-resources 
cost,  salaries  and  wages,  overhead,  etc. 

Cont  racts 

A  common  technique  that  is  used  to  specify  goals  and  tradeoffs  in 
military  system  development  is  the  multiple  incentive  contract.  The 
final  state  attributes  that  are  included  are:  some  measure,  or  meas¬ 
ures,  of  system  performance;  the  time  required  to  complete  development; 
and  the  amount  of  money  spent.  The  contract  specifies  for  various  com¬ 
binations  of  performance,  time,  and  cost  the  fee  the  contractor  will 
re  cei  ve . 

The  general  form  of  the  contracts  considered  in  this  study  is  given 
by 

TIP  =  CP  +  TP  +  PP*  (1) 

* 

The  symbols  used  here  are  those  used  in  the  FORTRAN  program  with 
the  exception  that  some  quantities  symbolized  in  this  study  are  not  used 
in  the  program. 


-10- 


where  TIP  -  the  total  incentive  payoff, 

CP  =  the  cost  payoff, 

TP  =  the  time  payoff, 

PP  =  the  performance  payoff. 

The  cost  term  determines  the  portion  of  the  total  cost  paid  by  the 

contractor;  hence,  it  is  not  a  "payoff”  in  the  positive  sense.  In  this 

* 

work  an  80-20  cost-sharing  ratio  is  used.  Suppose  that  the  target 
cost  is  $1  billion.  Then,  if  the  actual  cost  is  $1  billion  or  anything 
less,  the  contractor  pays  20  percent  of  the  cost  and  the  customer  pays 
80  percent.  If  the  actual  cost  is  more  than  the  target  cost,  the  con¬ 
tractor  pays  20  percent  of  $1  billion  plus  100  percent  of  the  cost  above 
it,  and  the  customer  pays  80  percent  of  $1  billion.  The  cost  payoff 
has  the  general  form 


[-.204, 


if  CA  £  CR , 


CP  = 


(2) 


-  CA  +  .80?,  if  CA  >  CR, 


where  CA  =  the  actual  cost  at  the  end  of  the  project, 
CR  =  the  "target"  cost. 

The  time  payoff  has  the  general  form 


TP 


RT(TR  -  TA) ,  if  TR  ^  TA, 


[-CT(TA  -  TR),  if  TR  <  TA, 


(3) 


where  RT  =  the  amount  of  reward  per  unit  of  time  for  delivery  before 
the  target  date, 

CT  =  the  amount  of  penalty  per  unit  time  for  delivery  after  the 
target  delivery  date, 

TA  =  the  actual  delivery  date, 

TR  =  the  target  delivery  date. 


See  Department  of  Defense  Incentive  Contracting  Guide,  U.S.  Gov¬ 
ernment  Printing  Office,  Washington,  D.C.,  1965,  pp.  16-18.  Note  that 
the  scheme  used  in  the  present  study  does  not  use  a  maximum  and  mini¬ 
mum  fee  limit. 


.'N 
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“V,v 


The  performance  payoff  consists  of  two  parts,  one  for  each  of  the 
two  system  performance  characteristics;  hence,  PP  can  be  written  as 


PP  =  PPNMI  +  PPGTO 


(4) 


where  PPNMI  =  the  performance  incentive  for  fuel  economy, 

PPGTO  =  the  performance  incentive  for  gross  take-off  weight. 
The  fuel  economy  incentive  is  given  by 


RPSNMKPSANMI  -  PSRNMI),  if  PSANMI  *  PSRNMI 


(5) 


[- CPS N MI ( PSRNMI  -  PSANMI),  if  PSANMI  <  PSRNMI, 


where  RPSNMI  =  the  reward  per  unit  of  performance  for  fuel  economy 
greater  than  the  target  fuel  economy, 

CPSNMI  =  the  penalty  per  unit  of  performance  for  fuel  economy 
less  than  the  target  fuel  economy, 

PSANMI  =  the  actual  fuel  economy  as  measured  by  the  test  procedure 
agreed  to  when  the  contract  was  made, 

PSRNMI  =  the  target  fuel  economy. 

The  gross  take-off  weight  incentive  is  given  by 


RPSGTO(PSRGTO-~  PSAGTO) ,  if  PSRGTO  >  PSAGTO 


PPGTO 


(6) 


rCPSGTO(PSAGTO  -  PSRGTO) ,  if  PSRGTO  <  PSAGTO 


where  RPSGTO  =  the  reward  per  unit  of  weight  for  gross  take-off  weight 
less  than  the  target  value, 

CPSGTO  =  the  penalty  per  unit  weight  for  gross  take-off  weight 
greater  than  the  target  value, 

PSAGTO  ”  the  actual  gross  take-off  weight  as  measured  by  the  test 
procedure  agreed  to  when  the  contract  was  made, 

PSRGTO  =  the  target  gross  take-off  weight. 


J 


-12- 


The  contract  is  thus  specified  by  giving  values  to  the  quantities 
CRy  RTy  CTy  TRy  RPSNMIy  CPSNMIy  PSRNMIy  RPSGTO ,  CPSGTOy  and  PSRGTO. 

Values  of  these  quantities  for  the  contracts  considered  in  the  study 
are  shown  in  Table  1. 

Con tra c ts_  Selected  for  Study 

Contract  NEUT  X  10  (also  labeled  Contract  B)  was  selected  for  use 
in  the  large-sample  runs  because  the  preliminary  results  indicated  that 
longer  searches  for  technically  superior  systems  were  experienced  with 
this  contract.  Contract  GTO  +  1  (also  labeled  Contract  A)  was  selected 
because  more  different  system  designs  were  selected  with  this  contract 
than  with  any  of  the  others.  The  rate  of  incentive  on  system  perform¬ 
ance  for  Contract  NEUT  X  10  is  ten  times  as  much  as  for  Contract 
CTO  +  1#  Hence,  Contract  NEUT  X  10  will  be  referred  to  as  the  high  in¬ 
centive  contract  and  Contract  CTO  4-  1  will  be  referred  to  as  the  low- 
incentive  contract. 

Relation  to  Utility  Functions 

Figures  1  and  2  show  the  lines  of  constant  performance  incentive 

payoff  for  Contracts  A  and  B.  To  anyone  that  is  familiar  with  the  in- 

* 

difference  curve  used  in  utility  theory,  the  parallel  between  multiple 
incentive  contracts  and  utility  functions  for  multi-attributed  alterna¬ 
tives  should  be  obvious.  In  fact,  the  only  difference  is  the  spacing 
between  the  lines  of  constant  payoff  and  the  corresponding  lines  of 
constant  utility.  There  are  three  distinguishable  cases.  First,  if 
payoff  equals  utility,  then,  the  utility  map  and  the  payoff  map  are  iden¬ 
tical.  Second,  if  the  utility  for  money  is  a  linear  function,  then,  the 
ratio  of  the  distances  between  curves  of  constant  payoff  and  the  cor¬ 
responding  utility  curves  will  be  a  constant.  This  constant  will  be 
equal  to  the  inverse  of  the  slope  of  the  utility  for  money.  The  third 
case  is  a  nonlinear  utility  for  money.  In  this  case,  for  a  payoff  map 


See  George  J.  Stigler,  The  Theory  of  Pricey  rev.  ed. ,  The  Macmillan 
Company,  New  York,  1952,  Chap.  5. 
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1  —  Constant  performance  incentive  Fig-  2--Constant  performance  incentive 

payoff  lines  for  Contract  A  payoff  lines  for  Contract  B 
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that  has  curves  separated  by  a  constant  amount,  the  corresponding  util¬ 
ity  curves  will  get  closer  together  or  farther  apart  depending  on  the 
local  value  of  the  second  derivative  of  the  utility  for  money. 


ALTERNATIVES 


In  almost  all  system  development  projects  there  will  be  more  than 
one  possible  design  for  at  least  one  part  of  the  system.  As  the  vari¬ 
ous  designs  for  the  components  are  investigated  through  the  engineering 
activities,  it  is  learned  which  designs  are  "best. 11  The  determination 
of  the  "best"  design  for  all  the  components  rests  on  simultaneous  con¬ 
sideration  of  the  effects  of  all  the  components  on  total  system 
performance . 

Call  the  parts  of  the  system  "components,"  and  suppose  the  sys¬ 
tem  consists  of  n  components.  Suppose  that  each  component  c ^  has  m \ 
alternative  (competing)  designs.  Denote  the  jth  alternative  for  com¬ 
ponent  i  by  c..  .  One  possible  system  design  would  be  c, , ,  ,  ..., 

tj  11  zi 

c '  ^  .  This  system  design  consists  of  the  number  "1"  design  for  each 
component.  Call  each  such  system  design  a  "combination,"  then  there 
are 


i=n 

JJ 

i=l 


m . 

t 


possible  combinations  for  the  system  (assuming  that  none  of  the  com¬ 
binations  are  infeasible) . 

A  combination  is  distinguished  from  a  "configuration"  because 
a  configuration  is  concerned  not  only  with  which  component  designs  are 
present,  but  where  they  are  located  with  respect  to  each  other.  For  ex¬ 
ample,  the  wing  on  airplanes  can  be  positioned  high  on  the  fuselage, 
low  on  the  fuselage,  or  somewhere  in  between.  Each  position  results 
in  a  different  configuration. 

In  the  present  study,  position  will  not  be  considered  as  an  ele¬ 
ment  of  choice  in  the  decision  process.  This  assumes  that  the  "best" 
relative  position  for  each  component  has  been  previously  determined. 
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In  the  hypothetical  development  program,  the  aircraft  is  consid¬ 
ered  as  having  three  major  subsystems:  the  engines,  the  airframe,  and 
all  other  subsystems — electronics,  hydraulics,  stores,  fuel,  crew,  etc. 
The  airframe  is  further  broken  down  into  two  major  components:  the 
fuselage  and  tail,  and  the  wing  and  nacelles.  This  breakdown  is  shown 
in  Fig.  3. 


Fig .  S — Subsystem  and  component  breakdown  of  aircraft 

The  subsystems  for  which  alternative  designs  are  available  are  the 
engines  and  the  airframe.  All  items  in  the  "Other"  category  are  as¬ 
sumed  to  be  off-the-shelf  items.  There  are  4  airframe  alternatives. 

The  model  of  the  system  development  process  simulates  the  development 
of  the  airframe  from  4  wing  designs  and  1  fuselage  design. 

The  development  project  thus  consists  of  the  development  of  an  air¬ 
frame  that  is  to  carry  a  specific  payload,  crew,  and  fuel  load  and  the 
selection  of  an  engine  to  go  with  the  airframe. 

ACTIVITIES 

The  activities  in  a  system  development  project  may  be  classified 
in  the  following  categories: 
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o  Building  models. 

o  Building  the  real  item. 

o  Buying  information  from  an  outside  source. 

The  first  category  includes  physical  models,  such  as  breadboards,  mock- 
ups,  scale  models,  etc.;  and  "paper"  models,  such  as  stress  analyses, 
simulations,  etc.  In  the  present  research,  analysis  of  the  informa¬ 
tion  obtained  from  such  activities  is  included  in  the  decisionmaking 
process  as  described  in  the  next  section. 

For  the  purpose  of  the  present  research,  the  activities  are  clas¬ 
sified  into  the  following  categories  and  are  detailed  later: 

o  Terminal  activities .  Deliver  the  best  of  any  previously  assem¬ 
bled  system. 

o  Assemble  and  test  activities.  Build  the  entire  system  (for  a 
given  configuration)  and  test  it. 

o  Experiment  activities .  Build,  test,  and  evaluate  a  mode]  of 
the  system  or  of  some  part  of  the  system. 

Buying  information  from  an  outside  source  will  be  ignored  in  the  pres¬ 
ent  study  because  it  requires  some  model  of  the  state  of  knowledge  at 
various  times  available  from  an  outside  source. 

Each  "experiment"  is  characterized  by  (1)  the  object  of  the  ex¬ 
periment  (i.e.,  a  part,  component,  etc.)  and  the  characterist ic(s) 
measured;  (2)  a  cost  to  perform,  assumed  to  be  known  from  past  experi¬ 
ence  and  independent  of  the  values  of  the  characteristics  of  the  object; 
(3)  a  time  to  perform,  also  assumed  known  and  independent  of  the  values 
of  the  characteristics;  and  (A)  the  effect  the  action  will  have  in  de¬ 
termining  a  new  variance  for  each  characteristic  measured,  also  assumed 
known. ^  The  new  variance  will  be  called  the  posterior  variance. 


A  model  may  be  full-scale.  It  is  distinguished  from  the  real 
item  by  whether  it  incorporates  all  features  of  the  real  item. 

The  effect  of  the  action  on  the  new  variance  is  expressed  by  the 
coefficient  of  variation.  Hence,  an  action  is  said  to  measure  some 
quantity  with  a  precision  of  plus  or  minus  some  percent.  This  proce¬ 
dure  is  at  least  partially  justified  in  the  present  research  because 
a  major  portion  of  system  development  is  engineering  and  in  engineer¬ 
ing  the  precision  of  an  experiment  is  fairly  well  known  ahead  of  time. 

*Note  that  the  prior  distributions  are  characteristics  of  the  ob¬ 
jects  of  the  action,  as  described  later  in  the  text. 
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These  four  elements  constitute  an  implicit  definition  of  an  "experi¬ 
ment"  action. 

Each  "assemble  and  test"  is  similarly  characterized  with  one  major 

difference;  the  expected  new  variances  for  all  characteristics  are 

zero,  because,  when  the  object  is  tested,  the  values  of  its  characteris- 

* 

tics  will  be  known  with  certainty  (f6r  the  test  conditions). 

Terminal  actions  are  characterized  by  a  time  and  cost  to  complete 
the  delivery.  The  system  performance  characteristics  are  known  with 
certainty  from  the  corresponding  assemble  and  test  (for  the  test 
conditions) . 

The  objects  of  the  activities  are  the  parts  of  the  system  and  the 
whole  system.  When  an  activity  is  applied  to  a  given  alternative  de¬ 
sign  of  some  part  of  the  system,  the  activity-object  pair  will  be  called 

an  action . 

For  the  hypothetical  development  project  used  in  this  research, 
only  a  small  portion  of  the  activities  involved  in  the  development  of 
an  aircraft  is  considered.  The  two  performance  characteristics  that 
are  used  in  the  multiple  Incentive  contract  served  as  a  guide  to  re¬ 
ducing  the  number  of  activities  to  be  included.  Activities  of  inter¬ 
est  are  those  that  are  concerned  with  the  aerodynamic  characteristics 
and  the  weight  of  the  aircraft. 

The  activities  are  further  categorized  by  the  part  of  the  system 
to  which  they  are  applied.  Hence,  the  weight-related  activities  include 
weight  estimation  for  fuselage  and  tail,  weight  estimation  for  wing  and 
nacelles,  stress  analysis  for  fuselage  and  tail,  stress  analysis  for 
wing  and  nacelles.  The  aerodynamic-related  activities  include:  wind 
tunnel  test  of  a  small-scale  model  of  the  entire  aircraft,  wind  tunnel 
test  of  a  large-scale  model  of  the  entire  aircraft,  wind  tunnel  test  of 
a  large-scale  model  of  the  wing  and  nacelles,  and  wind  tunnel  test  of 
a  large-scale  model  of  the  fuselage  and  tail  with  stub  wings  for  inter¬ 
ference  effects.  Two  additional  activities  are  required  to  obtain  the 
finished  product  and  to  terminate  the  process:  assemble  and  test  the 

; k 

The  test  conditions  are  important  because  they  will  be  the  condi¬ 
tions  used  to  determine  the  values  of  the  characteristics  of  the  final 
product . 
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comp  let  e  aircraft,  and  deliver  the  aircraft  to  the  customer.  The  time, 
cost,  and  precision  figures  are  summarized  in  Table  2.  The  activities 
are  briefly  described  as  follows: 


Table  2 

SUMMARY  OF  ACTION  CHARACTERISTICS 


Activity 

Object 

Characteristic ( e ) 
Measured 

Time 

(tik8) 

Cost 

($) 

Precision 

(X) 

No.  of  Pos¬ 
sible  Actions 
and  ID  Nos . 

De  1  i  ve  r 

Complete 

aircraft 

— 

1 

1,000 

— 

8,  nos. 

1-8 

Assemble  and 

test 

Complete 

aircraft 

Fuel  economy, 
gross  take-off 
weight 

52 

300,000,000 

0 

8,  nos. 

9-16 

Wind  tunnel, 
large  air¬ 
craft 

Complete 
aircraf  t , 
including 
engines 

Drag  coefficient 
for  aircraft, 
efficiency 
factor 

20 

3,000,000 

7 

8,  nos. 
17-24 

Wind  tunnel, 
small  air¬ 
craft 

Aircraft 

excluding 

engines 

Drag  coefficient 
for  airframe, 
efficiency 
factor 

5 

160,000 

10 

4,  nos. 
25-28 

Weight 

es  timat ion 

Wings  and 
nacelles 

Weight  of  wing 
and  nacelles 

1 

5,000 

10 

4  ,  nos . 
29-32 

Wind  tunnel, 
large  wing 

Wings  and 
nacelles 

Drag  coefficient 
for  winjj  and 
nacelles,  effi¬ 
ciency  factor 

10 

500,000 

3 

4,  nos. 
33-36 

Stress 

analysis 

Wings  and 
nacelles 

Weight  of  wing 
and  nacelles 

16 

500,000 

3 

4 ,  nos . 

3  7-40 

Weight 

estimation 

Fuselage 
and  tail 

Weight  of  fuse¬ 
lage  and  tall 

1 

5,000 

10 

1 ,  no . 

41 

Wind  tunnel, 
large 
fuselage 

Fuselage 
and  tall 

Drag  coefficient 
for  fuselage 
and  tall 

10 

500,000 

3 

1 ,  no . 

42 

Stress 

analysis 

Fuselage 
and  tail 

Weight  of  fuse¬ 
lage  and  tall 

10 

200,000 

3 

1 ,  no . 

43 

NOTE:  The  figures  have  been  derived  from  estimates  made  by  professionals  who  have 

been  involved  in  aircraft  development.  The  final  numbers  were  determined  by  adjusting 
the  individual  estimates  so  that  the  entire  set  was  "reasonable"  from  both  an  absolute 
and  a  relative  point  of  view. 

The  measure  of  precision  used  in  this  research  is  the  coefficient  of  variation,  which 
is  equal  to  the  standard  deviation  divided  by  the  mean.  The  figures  are  expressed  as  per- 
centages  in  the  table. 
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o  Weight  Estimation .  The  weight  estimation  activity  envisaged 
in  this  study  involves  about  5  technical  people  and  requires 
approximately  1  week  of  time.  The  figures  for  time,  cost,  and 
precision  are:  (1)  time  equals  1  week;  (2)  cost  equals  $5000; 
and  (3)  precision  equals  10  percent  of  the  estimate.  (Precision 
refers  to  the  estimate  of  the  weight  of  the  structural  member 
only  and  does  not  include  electrical,  hydraulic,  or  any  other 
equipment  weights.) 

o  Stress  Analysis .  The  stress  analysis  activity  involves  more. 
Hundreds  of  people  are  involved  over  a  longer  period  of  time 
and  the  precision  of  the  resulting  weight  figure  is  much  better. 
The  purpose  of  the  stress  analysis  is  to  determine  how  the  air¬ 
frame  is  to  be  built  so  that  it  will  be  structurally  sound. 

Once  a  sound  design  is  worked  out,  the  weights  of  all  cubic 
inches  of  material  are  added  up  to  get  the  total  weight.  The 
time  required  for  stress  analysis  is  10  weeks.  The  cost  for 
the  fuselage  and  tail  is  $200,000  and  for  the  wing  and  nacelles 
it  is  $500,000.  The  cost  for  the  wing  analysis  is  higher  be¬ 
cause  increased  technical  effort  is  required.  The  precision 
of  the  weight  obtained  is  3  percent. 

•k 

o  Wind  Tunnel:  General .  Wind-tunnel  testing  generates  informa¬ 
tion  about  many  aerodynamic  characteristics  of  an  aircraft. 

The  many  detailed  measurements  are  used  to  evaluate  the  aircraft’s 
lift-drag  performance,  its  stability  and  control  characteris¬ 
tics,  and  its  general  handling  qualities.  In  this  study  only 
the  lift-drag  performance  is  of  interest;  however,  the  times 
and  costs  reflect  the  total  job. 

To  carry  out  a  wind-tunnel  test,  it  is  necessary  to  plan 
the  test(s),  build  the  model,  obtain  the  use  of  a  wind  tunnel, 
run  the  test(s),  and  analyze  the  data. 

o  Wind  Tunnel:  Small  Aircraft .  The  wind-tunnel  testing  of  a 
small  model  of  the  aircraft  uses  a  model  with  a  5-ft  wing  span. 

•k 

The  reader  is  referred  to  Alan  Pope,  Wind-Tunnel  Testing,  2d  ed.  , 
John  Wiley  &  Sons,  Inc.,  New  York,  1954,  for  a  thorough  explanation  of 
wind-tunnel  testing. 
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The  time,  cost,  and  precision  figures  are:  time  equals  5  weeks; 
cost  equals  $160,000;  and  precision  equals  10  percent.  Infor¬ 
mation  obtained  is  for  the  frictional  drag  coefficient  of  the 

aircraft,  Cr  ,  and  the  airplane  efficiency  factor,  e. 

L  f 

Wind  Tunnel:  Large  Aircraft .  The  model  used  in  this  test  has 
a  wing  span  of  approximately  10  feet.  The  model  has  remotely 
operable  flaps,  landing  gear,  control  surfaces,  bomb  bays  and 
electrically  driven  propellers.  The  test  provides  information 
about  >  and  £•  The  time  required  is  20  weeks,  the  cost  is 
$3  million,  and  the  precision  is  7  percent. 

Wind  Tunnel:  Large  Wing .  The  model  used  in  this  test  has  a 
span  (single  wing)  of  20  feet.  The  information  obtained  is  for 
and  e  of  the  wing.  The  time  required  is  10  weeks,  the  cost 
is  $500,000,  and  the  precision  is  3  percent. 

Wind  Tunnel:  Large  Fuselage .  The  model  has  a  height  of  6  feet 
and  a  stub-wing  span  of  20  feet.  The  information  obtained  is 
for  Cp  of  the  fuselage  and  tail.  The  time  required  is  10  weeks, 
the  cost  is  $500,000,  and  the  precision  is  3  percent. 

Assemble  and  Test .  A  complete  testing  program  requires  construc¬ 
tion  of  three  aircraft:  one  is  for  static  test  and  two  are  for 
flight  test.  In  the  present  study,  the  results  of  assemble 
and  test  are  a  value  for  the  gross  take-off  weight,  a  value  for 
the  frictional  drag  coefficient  of  the  aircraft,  and  a  value 
for  the  airplane  efficiency  factor.  The  latter  two  figures  are 
used  to  compute  the  fuel  economy  figure.  The  time  for  assemble 

and  test  Is  52  weeks,  the  cost  is  $300  million,  and  the  results 

* 

are  perfect — no  uncertainty  remains. 

Deliver .  The  terminal  action  requires  only  a  minimum  amount  of 
paper  work.  The  customer  usually  sends  a  crew  to  pick  up  the 
airplane.  The  time  Is  1  week,  and  the  cost  is  $1000.  A  preci¬ 
sion  figure  does  not  apply.  The  terminal  action  is  concerned 


* 

In  reality,  there  will  always  be  some  residual  uncertainty.  How¬ 
ever,  the  results  obtained  in  this  study  are  not  compromised  by  this 
assumption.  The  only  thing  affected  by  this  assumption  is  the  testing 
procedure(s)  to  determine  the  system  performance  values  to  be  used  in 
the  incentive  contract,  and  the  procedure  is  not  made  explicit  in  this 
study , 
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only  with  time  and  cost,  since  all  performance  characteristics 

i 

have  been  determined  under  assemble  and  test. 

i 

? 

) 

STATE  OF  KNOWLEDGE;  TECHNOLOGY 

\ 

At  any  given  time  there  is  a  state  of  knowledge  regarding  the  sys- 
>  tern  being  developed.  This  knowledge  may  be  classified  into  two  cate¬ 
gories:  information  regarding  Values  of  the  characteristics  of  the 

system  and  its  parts  and  the  relationships  between  the  characteristics. 
Information  regarding  the  characteristics  will  be  discussed  in  the  next 
section . 

The  relationships  between  the  characteristics  are  a  body  of  sci¬ 
entific  or  technical  knowledge — termed  here  a  "technology.”  As  an  ex¬ 
ample,  the  weight  of  an  aircraft  must  be  the  sum  of  the  weights  of 
everything  that  goes  into  it.  Similarly,  but  not  as  simply,  the  over¬ 
all  performance  of  the  aircraft  is  influenced  by  the  performance  of 
the  engines  and  the  aerodynamic  characteristics  of  the  airframe.  These 
relationships  form  a  hierarchy  as  shown  in  Fig.  4. 


Fig.  4 — Hierarchy  of  design  equations  and  inputs 
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A  given  system  development  project  is  concerned  with  a  given  tech 
nology  or  group  of  technologies.  Development  of  a  turbine-engine  air¬ 
craft  is  concerned  with  the  technologies  of  turbine  engines  and  aero¬ 
dynamics  . 

To  be  amenable  to  the  present  research,  the  technologies  must  be 

represented  mathematically — by  analytical  functions,  graphs,  or  tables 

(see  Fig.  5).  It  is  not  necessary  that  these  representations  be  exact 

* 

however,  in  the  present  research  they  will  be  considered  to  be  so. 

CL 

Equation:  Cn  =  Cn  +  — — — 

D  Dj.  TT  R  e 

C  =  functional  drag  coefficient  ■  .0250 

f 

Al  85  aspect  ratio  =  7 
e  m  airplane  efficiency  factor  -  .88 


Table: 


CD 

.0250 

0.00 

.0290 

0.20 

.0330 

0.40 

.0370 

0.60 

.0410 

0.80 

.0450 

1.00 

Graph : 


Fig .  5 — Alternative  forme  of  presenting  the  relation  between 
the  lift  coefficient j  tJte  drag  coefficient ,  Cp3 

for  a  "typical"  aircraft 


An  interesting  area  for  further  research  is  concerned  specific¬ 
ally  with  uncertainty  about  the  functional  forms  of  technology — call 
it  scientific  uncertainty.  See  the  discussion  in  Sec.  VII y  under 
"Extensions . " 
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Fur  thermo  re,  in  the  present  research,  only  analytical  functions  will  be 
used;  tables  or  graphs  could  be  used  just  as  effectively  but  probably 
not  as  efficiently  because  the  memory  (core)  requirement  for  the  tables 
is  very  large.  The  analytical  functions  are  the  design  equations  re¬ 
ferred  to  previously. 

For  the  hypothetical  development  project  the  equations  are  con¬ 
cerned  with  weight  and  aerodynamic  characteristics.  The  fuel  economy 
performance  of  the  aircraft  is  the  result  of  the  combined  aerodynamic 
performance  of  the  aircraft  and  the  performance  of  the  engines.  The 
gross  take-off  weight  is  the  simple  sum  of  the  weight  of  all  parts  of 
the  aircraft.  Derivation  of  the  fuel  economy  equations  is  presented 
in  Appendix  A. 

STATE  OF  KNOWLEDGE:  HIERARCHY  OF  UNCERTAINTY 

The  state  of  knowledge  regarding  the  values  of  the  system  and  com¬ 
ponent  characteristics  at  any  given  time  can  be  represented  by  proba¬ 
bility  distributions.  The  uncertainty  regarding  the  system  character¬ 
istics  (the  aggregate  technological  uncertainty)  is  related  by  the 
design  equations  to  the  uncertainty  regarding  the  component  character¬ 
istics.  Hence,  there  is  a  hierarchy  of  uncertainty  as  shown  in  Fig.  6. 

The  distributions  at  the  bottom  levels  represent  knowledge  gained 
by  performing  engineering  tasks  such  as  stress  analyses,  wind-tunnel 
tests,  weight  estimates,  etc.  The  dispersion  of  these  distributions 
depends  on  the  precision  associated  with  the  particular  task.  For  ex¬ 
ample,  a  weight  estimate  by  a  qualified  engineer  may  be  good  to  plus 
or  minus  10  percent,  while  the  weight  determination  derived  from  a  com¬ 
plete  structural  stress  analysis  may  be  good  to  plus  or  minus  2  percent. 
The  distributions  at  the  intermediate  and  top  levels  represent  the  un¬ 
certainty  propagated  from  the  bottom  level. 

Two  methods  can  be  used  to  determine  the  aggregate  uncertainty  for 
any  given  situation:  Monte  Carlo  Simulation  and  the  Propagation  of 
Error.  These  two  techniques  have  different  characteristics,  and  the 
problem  of  deciding  which  to  use  in  any  given  situation  is  discussed 
in  Appendix  D.  For  the  example  used  in  the  present  research,  the  means 
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Fig.  G — Hierarchy  of  uncertainty 

and  standard  deviations  obtained  from  the  two  methods  are  reasonably 
close.  Therefore,  because  propagation  of  error  requires  much  less  com¬ 
puter  time  and  because  the  purposes  of  this  study  require  many  computer 
runs,  the  propagation  of  error  technique  was  favored  for  use  in  this 
study.  This  technique  requires  that  the  design  equations  be  expressed 
in  analytical  form. 

For  the  hypothetical  development  project,  there  are  eight  different 
hierarchies.  Figure  7  shows  the  initial  hierarchies  for  all  eight  sys¬ 
tem  configurations.  The  figures  at  the  lowest  level  of  each  branch 
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NOT  reproducible 


Fig.  7 — Initial  hierarchy  of  uncertainty 

* 

were  specified  first.  Then  the  figures  at  higher  levels  were  deter¬ 
mined  using  the  design  equations  relating  the  characteristics  at  adja¬ 
cent  levels  and  the  propagation  of  error  technique. 

The  characteristics  that  describe  an  engine  are  the  coefficients 
and  constant  terms  of  equations  that  are  developed  in  Appendix  A.  En¬ 
gine  alternative  E ^  was  determined  using  the  T56-A-10W/WA  Model 

* 

The  mean  values  were  obtained  by  using  the  P-3  aircraft  and 
T-56-A-10W  engine  data  and  generating  alternatives  based  on  suggestions 
by  personnel  at  The  Rand  Corporation  and  Lockheed  California  Company. 
The  standard  deviations  were  based  on  estimates  from  the  same  people. 
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Specification.  To  insure  that  there  would  be  a  trade-off  between  en¬ 
gine  E ^  and  engine  E^>  the  second  engine  was  selected  to  offer  better 
fuel  economy  bOt  at  a  higher  weight.  The  values  for  engine  E g  were 
determined  by  increasing  the  weight  above  the  weight  of  E ^  and  then 
testing  the  effect  of  small  changes  in  the  coefficients  and  constant 
terms  just  mentioned  until  an  interesting  set  of  numbers  was  found. 

The  characteristics  that  describe  the  fuselage  and  tail  are  ap¬ 
proximate  figures  for  the  Lockheed  P-3A  Orion  Aircraft.^ 

The  characteristics  that  describe  alternative  wing  design  W  are 

*  A 

also  approximate  figures  for  the  P-3A.  The  other  wings  were  deter¬ 
mined  by  varying  the  wing  area,  S,  and  the  aspect  ratio,  /R  ,  from  the 
nominal  P-3A. 

The  items  in  the  "Other11  category  contribute  only  weight  from  the 
point  of  view  of  the  present  study.  The  weight  figure  used  is  approx¬ 
imately  the  weight  of  the  "Other"  items  for  the  P-3A. 

The  initial  uncertainties  were  established  by  considering  what 
might  be  reasonable  and  how  much  would  be  interesting.  Several  test 
runs  of  the  program  were  made  with  different  initial  uncertainties  be¬ 
fore  the  values  shown  in  Fig.  7  were  adopted. 


Initially,  all  engine  parameter  values  were  changed  by  the  same 
percentage.  This  resulted  in  such  a  drastic  change  in  the  fuel  economy 
performance  that  the  method  of  small  perturbations  was  used. 

^Personal  communications  from  E.C.B.  Danforth  and  D.  Beier  of  Lock 
heed  California  Company. 

± 

Ibid. 

"k"k 

Mr.  Danforth  (see  previous  footnotes)  suggested  that  optimiza¬ 
tion  of  the  aircraft  design  with  respect  to  fuel  economy  would  probably 
result  in  the  selection  of  a  higher  aspect  ratio  wing;  hence,  the  direc 
tion  of  change.  Of  course,  the  P-3A  did  not  have  a  higher  aspect  ratio 
wing,  but  there  were  other  trade-offs  involved  in  the  P-3  development. 
Furthermore,  that  development  was  unique  in  that  the  airframe  had  al¬ 
ready  been  developed  and  produced  as  a  commercial  airliner — the  Electra 
Consequently,  very  few,  if  any,  comparisons  can  be  made  between  the  P-3 
development  and  this  study.  The  P-3  was  used  here  to  insure  that  the 
aircraft’s  characteristics  were  consistent  and  realistic.  For  a  brief 
account  of  the  P-3  development,  the  reader  should  consult  Jane's  All 
The  World's  Aircraft,  Sampson  Low  Marston  &  Co. ,  Ltd. ,  London.  The 
account  in  the  1963-1964  edition  is  the  most  complete  regarding  the 
early  development . 
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The  initial  uncertainties  regarding  the  fuel  economy  and  the  gross 
take-off  weight  are  shown  in  Fig.  8.  The  diagonal  lines  in  Fig.  8  are 
the  zero  performance  incentive  payoff  lines.  The  vertical  and  hori¬ 
zontal  dashed  lines  divide  the  technical  performance  plane  into  four 
quadrants  labeled  I,  II,  III,  and  IV.  The  origin  of  this  division  is 
at  the  zero-incentive  performance  values:  .088  n  mi/lb  and  130,000  lb. 
Outcomes  in  quadrant  IV  are  superior  to  both  zero-incentive  values. 

Fuel  economy  is  higher  and  gross  take-off  weight  is  lower.  In  quad¬ 
rant  I,  the  aircraft  is  overweight  but  the  fuel  economy  is  superior  to 
the  zero-incentive  value.  In  quadrant  III,  the  aircraft  is  inferior 
to  the  zero-incentive  fuel  economy  value,  but  it  is  underweight.  In 
quadrant  II,  the  aircraft  is  inferior  with  regard  to  both  characteristics. 

The  probability  density  functions  for  the  initial  fuel  economy  un¬ 
certainties  and  the  initial  gross  take-off  weight  uncertainties  are 
shown  in  Figs.  9  and  10,  respectively. 


Gross  take-off  weight  (lb) 
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Fig.  8-- Initial  system  performance  uncertainties 


Probability  per  unit  of  gross  take-off  weight  (x  1  0  )  Probability  per  unit  of  fuel  economy 
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Fuel  economy  (n  mi/lb) 

Fig.  9 — Initial  fuel  economy  uncertainties 


III.  THE  MANAGEMENT  CONSIDERATIONS  IN 


A  SYSTEM  DEVELOPMENT  PROJECT: 

A  SIMULATION  MODEL  OF  SYSTEM  DEVELOPMENT  DECISIONMAKING 


GENERAL 

At  various  times  during  the  course  of  a  system  development  proj¬ 
ect,  a  decision  must  be  made  about  what  to  do  next.  These  decisions 
and  policies  for  making  them  are  the  main  focus  of  this  work.  The 
following  is  a  rough  description  of  the  activities  and  events  involved: 

1.  A  decision  is  made  about  what  to  do  next. 

2.  The  activity  is  carried  out. 

3.  Information  resulting  from  the  activities  become  available. 

A.  The  consequences  of  the  information  are  determined. 

5.  A  decision  is  made. 

etc. 

A  generalized  model  of  this  process  is  shown  in  Fig.  11.  This  diagram 
represents  the  major  decision  and  information  functions  of  the  process 
of  system  development.  The  only  real  difference  between  the  diagram 
and  the  actual  computer  program  used  in  this  study  is  that  the  bookkeep¬ 
ing  functions  are  not  represented  in  the  diagram.  The  discussion  in 
this  section  will  proceed  using  Fig.  11  as  an  outline.  A  more  detailed 
description  of  the  program  is  contained  in  Appendix  C. 

INITIALIZATION 

Evaluation  of  alternative  decisionmaking  policies  by  repeated 
simulation  of  the  decisionmaking  process  requires  that  each  time  the 
process  is  begun,  the  conditions  must  be  the  same.  In  the  present  re¬ 
search,  the  initial  conditions  of  importance  are  the  initial  state  of 
knowledge — both  the  technology  and  the  initial  distributions  for  the 
characteristics  of  the  system  and  its  parts — and  the  characteristics 
of  the  actions. 
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Select  action 

Execute  action  and 
receive  information 
about  component 
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Fig .  11 — General  model  of  system  development  decision 
and  information  elements 


In  the  present  research,  the  technology  and  the  characteristics 

k 

of  the  actions  remain  constant.  The  technology  is  represented  by  the 
design  equations.  The  characteristics  of  the  actions  are  read  in  as 
data  and  can  be  changed  with  no  reprogramming. 

The  distributions  for  the  component  and  system  characteristics 
change  when  actions  are  executed.  Hence,  they  must  be  restored  to 
their  initial  values  each  time  a  new  simulation  is  begun. 


DECISIONMAKING  POLICY 


A  decisionmaking  policy  consists  of  operations  that  identify  eligi¬ 
ble  actions;  forecast  the  time,  cost,  and  performance  information  to  be 


•k 

For  future  use,  it  would  be  desirable  to  relate  the  times  and  costs 
of  the  actions  to  the  characteristics  of  the  component  or  subsystem  in¬ 
volved  in  the  actions.  See  section  on  refinements  in  Sec.  VII. 

The  author’s  previous  work  on  measuring  progress  in  system  devel¬ 
opment  is  concerned  with  such  changes  in  probability  distributions.  See 
Timson,  op.  cit. 
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used  in  evaluation;  evaluate  actions  using  the  forecasts;  select  the 
action  that  maximizes  or  minimizes  the  evaluation  criterion;  and  exe¬ 
cute  that  action.  These  operations  will  be  discussed  in  general  terms 
immediately  below.  Specific  details  of  the  policies  studied  in  this 
research  are  presented  in  Sec.  IV. 

Eligibility 

As  was  mentioned  in  Sec.  I,  eligibility  is  related  to  scheduling. 
Scheduling  analysis  developed  in  production  management  and  was  concerned 
with  questions  of  the  logical  sequencing  of  actions  and  the  availabil¬ 
ity  of  and  location  of  resources  required  to  execute  the  actions. 

In  system  development,  and  any  other  process  that  involves  learn¬ 
ing,  another  sequencing  consideration  becomes  important.  This  can  be 
phrased  as  a  question:  Are  there  any  actions  with  particular  charac¬ 
teristics  that  should  be  executed  at  some  particular  decision  point  or 
in  a  particular  sequence?  This  consideration  is  concerned  with  whether 
there  are  actions  that  can  obtain  some  particularly  useful  information 
early  in  the  program,  or  that  can  obtain  some  information  at  very  low 
cost  and  in  a  very  short  time,  or  that  can  obtain  information  that  will 
be  more  useful  when  additional  information  is  available.  An  example  of 

such  a  consideration  is  to  perform  all  actions  that  quickly  and  cheaply 

* 

will  yield  information  on  many  alternative  designs  or  approaches. 

In  the  present  research,  the  sequences  of  actions  is  due  more  to 
the  step-by-step  evaluation  and  selection  of  actions  than  to  any  pre¬ 
determined  scheduling;  hence,  the  term  "eligibility"  is  used  to  identify 
the  sequencing  or  scheduling  aspects  of  the  process.  At  each  decision 
point,  all  actions  are  screened  to  see  if  they  satisfy  the  sequencing 
considerations.  Those  actions  that  do  are  said  to  be  "eligible"  for 
selection.  These  actions  are  evaluated  after  information  regarding 
their  characteristics  and  the  present  state  of  knowledge  regarding  the 
project  are  subjected  to  a  forecast,  as  described  next. 

* 

This  corresponds  to  one  of  the  "rules  of  a  good  development  pol¬ 
icy"  prescribed  by  Klein,  op.  cit.,  pp.  4-5. 
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Fo  re  cast  Information 

Given  a  list  of  eligible  actions  to  evaluate,  it  is  necessary  to 
have  the  ir  formation  to  be  used  in  the  evaluation  and  a  rule  for  proces¬ 
sing  the  information.  The  model  of  system  development  used  here  con¬ 
tains  a  number  of  items  that  can  be  used  in  the  evaluation.  These  are 
as  follows: 

o  The  state  of  knowledge,  which  may  be  the  present  state  or  a 
* 

lagged  state. 

o  The  characteristics  of  the  actions,  including  times,  costs, 
and  precision  of  new  knowledge. 

o  The  time  and  money  consumed. 

Any  combination  of  these  items  may  be  used. 

In  general,  the  evaluation  rule  will  be  based  on  some  forecasted 
values  for  the  items  considered.  The  available  information  regarding 
the  items  can  be  processed  in  a  number  of  ways  to  obtain  different 
forecasts  of  the  outcomes  that  will  result  from  actions.  Consequently, 
the  methods  of  forecasting  will  influence  the  decisions  that  are  made. 

In  the  present  study,  four  policies  have  been  examined  that  consider 
time,  cost  and  performance,  and  one  policy  that  considers  performance 
only . 

Time  and  Cost  Forecasts.  Forecasting  time  and  cost  may  be  based 
on  many  factors.  In  this  study,  one  forecasting  procedure  is  used  in 
all  policies  that  require  a  forecast  of  time  and  cost.  The  procedure 
makes  a  forecast  of  the  minimum  number  of  actions  required  to  complete 
the  project  as  imposed  by  the  eligibility  rule.  It  does  not  reflect 
uncertainty.  Details  of  the  procedure  are  described  in  Sec.  IV. 

Performance  Forecasts .  Information  regarding  technical  perform¬ 
ance  for  use  in  evaluation  may  also  be  determined  in  a  number  of 

* 

Time  lags  may  affect  all  information  or  only  certain  items  of  in¬ 
formation.  Examination  of  different  lags  and  different  patterns  of 
lags  gives  some  indication  of  the  value  of  different  communication 
patterns  in  a  system  development  organization. 
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different  ways.  Before  discussing  the  procedures  used  in  this  study, 

some  terms  must  be  defined  to  identify  the  probability  distributions 
* 

at  various  times.  The  probability  distribution  available  before  an 
information-buying  action  is  taken  is  called  the  prior  probability  dis¬ 
tribution,  or  simply  the  prior.  The  prior  specifies  the  distribution 
of  outcomes  that  would  be  expected  if  a  terminal  action  was  to  be  taken. 
The  probability  distribution  available  after  the  action  is  taken  and 
the  new  information  is  learned  is  the  posterior  probability  distribu¬ 
tion,  or  simply  the  posterior.  Before  the  information-buying  action 
is  taken,  a  forecast  can  be  made  of  the  distribution  of  the  mean  of  the 
posterior  if  the  sample  variance  is  known.  This  distribution  is  known 
as  the  prior  distribution  of  the  posterior  mean,  or  simply  the  preposte¬ 
rior.  The  preposterior  specifies  the  distribution  of  outcomes  that 
would  be  expected  if  the  information-buying  action  was  to  be  taken,  the 
new  information  learned,  and  the  terminal  action  taken. 

Three  performance  forecasting  schemes  are  included  in  the  policies 
considered  in  this  study.  The  simplest  scheme  sets  the  mean  of  the 
forecast  distribution  equal  to  the  prior  mean  and  the  standard  devia¬ 
tion  of  the  forecast  equal  to  zero;  i.e.,  the  prior  mean  is  used  as  a 
certainty  equivalent.  Because  the  probability  distributions  in  this 
study  are  all  assumed  to  be  normal,  the  preposterior  mean  is  equal  to 
the  prior  mean;  hence,  the  forecast  is  just  the  preposterior  mean. 

A  second  forecasting  scheme  uses  the  complete  preposterior  analy¬ 
sis. *  *  For  any  given  action,  the  precision  of  the  new  assessment  (after 
the  action)  is  known.  This  precision  is  related  to  the  sampling  vari¬ 
ance.  To  determine  the  preposterior  distributions  for  characteristics 
at  the  system  level,  first  identify  the  component  characteristics  whose 
estimated  values  will  be  refined  by  the  action  being  evaluated.  Then 
determine  the  preposterior  distribution  for  those  characteristics. 

Next,  using  the  preposterior  for  the  characteristics  whose  estimated 

* 

The  terminology  is  that  used  in  common  practice.  See,  for  exam¬ 
ple,  Robert  Schlaifer,  Probability  and  Statistics  for  Business  Deci¬ 
sions  y  McGraw-Hill,  New  York,  1959.  See  especially  p.  337  and  Chap.  33. 

tIbid. ,  pp.  525-528. 

*Loc.  cit. 
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values  will  be  refined  and  the  priors  for  all  other  (non-re fined)  char¬ 
acteristics,  determine  the  preposteriors  for  higher  level  characteris¬ 
tics  using  the  updating  procedures  described  below.  This  provides  a 
forecast  in  the  form  of  a  distribution  of  outcomes  that  may  result  from 
a  given  action. 

The  third  forecasting  scheme  considered  in  this  study  uses  the 
preposteriors  for  the  characteristics  measured  by  a  given  action.  Fore¬ 
casts  for  nigher  levels  are  made  by  applying  the  updating  procedures 
using  these  preposteriors  and  the  initial  prior  means  for  the  charac¬ 
teristics  not  affected  by  the  action.  The  other  characteristics  are 
represented  by  the  means  of  the  initial  prior  distributions  for  the 
particular  design  that  was  the  best  at  the  beginning  of  the  project  as 
if  no  new  information  was  communicated  during  the  project. 

Comparing  results  obtained  using  the  first  scheme  with  results 
using  the  second  scheme  gives  an  indication  of  the  value  of  doing  a 
complete  preposterior  analysis  using  all  available  information  concern¬ 
ing  uncertainty  as  compared  to  using  only  the  means.  Comparing  results 
obtained  using  the  second  scheme  with  results  using  the  third  scheme 
gives  an  indication  of  the  value  of  using  information  currently  avail¬ 
able  for  all  decisions  as  opposed  to  using  "old"  (unrevised)  informa¬ 
tion  regarding  all  items  except  those  concerned  with  the  immediate  action. 

EVALUATION  AND  SELECTION  RULES 

An  evaluation  rule  is  a  rule  that  prescribes  how  the  information 
that  results  from  the  forecast  will  be  combined  to  arrive  at  a  number 
that  indicates  the  value  of  carrying  out  an  action. 

In  this  study,  two  evaluation  rules  are  used.  The  first  is  an  ap¬ 
proximation  to  the  ’’standard"  prescription  of  normative  decision  theory — 

* 

The  rule  is  called  an  approximation  so  the  reader  will  not  be 
misled  into  thinking  that  the  decisionmaking  policies  used  in  this 
study  are  on  the  same  basis  as  the  expected  payoff  calculations  found 
in  sequential  decision  theory.  The  fundamental  difference  is  that  se¬ 
quential  decision  theory  involves  only  one  kind  of  information-buying 
action,  while  the  present  work  is  concerned  with  several  types  of  in¬ 
formation  buying  actions.  The  present  situation  is  further  complicated 
by  the  requirement  that  a  certain  number  of  information-buying  actions 
be  executed  before  a  terminal  action  is  permitted  (called  the  "eligi¬ 
bility  rule"  here).  The  calculations  in  sequential  decision  theory 


-37- 


calculate  the  expected  payoff  for  each  eligible  action  and  select  that 
action  that  maximizes  the  expected  payoff.  The  payoff  is  determined 
by  the  multiple  incentive  contract;  and  because  the  system  performance 
characteristics  are  represented  by  probability  distributions,  the  pay¬ 
off  must  also  be  represented  by  a  probability  distribution.  Utility 
for  money  is  not  considered  in  this  study. 

The  second  rule  is  to  calculate  the  probability  of  technical  suc¬ 
cess  for  each  eligible  action  and  select  that  action  that  maximizes 
the  probability  of  technical  success.  The  probability  of  technical  suc¬ 
cess  is  defined  as  the  probability  that  the  system  will  meet  or  exceed 
the  zero-incentive  levels  of  all  system  performance  characteristics  con¬ 
tained  in  the  incentive  contract. 

EXECUTION  AND  RECEIPT  OF  INFORMATION 

Once  an  action  has  been  selected,  execution  is  represented  by  a 
simple  random  process  that  determines  the  new  state  of  knowledge.  In 
addition,  many  bookkeeping  activities  are  performed.  Execution  of  the 
action  results,  in  the  real  world,  in  the  consumption  of  some  resources, 
the  passing  of  some  time,  and  the  gaining  of  some  knowledge.  In  the 
simulation  model,  the  consumption  of  resources  is  represented  by  incur¬ 
ring  the  cost  associated  with  the  action;  the  passing  of  time  is  rep¬ 
resented  by  the  time  associated  with  the  action;  and  the  gaining  of 
knowledge,  for  the  characteristics  measured  by  the  action,  is  deter¬ 
mined  by  drawing  a  random  sample  from  the  prior  distributions.  The 
means  of  the  new  distributions  are  set  equal  to  the  sample  means  from 
the  respective  distributions.  The  standard  deviations  of  the  new  dis¬ 
tributions  are  set  equal  to  the  product  of  the  sample  standard  devia¬ 
te 

tions  associated  with  the  actions  times  the  respective  new  means. 


compares  the  expected  value  of  taking  a  terminal  action  with  the  ex¬ 
pected  value  of  an  information-buying  action  followed  by  a  terminal 
action.  In  the  present  work,  a  terminal  action  is  not  always  permit¬ 
ted.  Furthermore,  the  fact  that  several  actions  may  be  required  by 
the  eligibility  rule  before  the  terminal  action  is  permitted  makes  it 
a  difficult  task  to  examine  all  permutations  and  combinations  of  ac¬ 
tions  required  to  reach  a  terminal  decision.  Thus,  the  evaluation  step 
is  based  on  comparing  individual  information-buying  actions  followed 
by  a  terminal  action,  whether  this  is  feasible  or  not.  Terminal  ac¬ 
tions  enter  the  evaluation  only  when  they  are  ’’eligible . M 

An  alternative  method  of  revising  probability  distributions  in 
the  light  of  new  information  is  the  ’’Bayesian”  method.  See  Schlaifer, 
op.  cit.,  Chap.  21. 
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The  bookkeeping  requirements  are  the  following: 

1.  Identify  the  selected  action  by  the  present  step  number. 

2.  Increase  the  step  number  by  one. 

3.  Eliminate  the  selected  action  from  future  consideration. 

4.  Eliminate  any  actions  preempted  by  the  choice  from  future 
consideration. 

5.  Record  any  prerequisites  satisfied  by  having  done  the 
selected  action. 

TEST  FOR  TERMINAL  ACTION 

The  test  for  selection  of  a  terminal  action  is  only  one  of  the 
many  branching  points  in  the  whole  program.  The  only  reason  it  is 
shown  in  Fig.  11  is  because  it  leads  to  termination  of  the  process. 

UPDATING  SUPERIOR  QUANTITIES 

If  the  action  selected  is  not  a  terminal  action,  then  the  compo¬ 
nent  and  system  characteristics  that  are  superior  to  those  measured 
must  be  revised  in  the  light  of  the  new  information. 

Updating  superior  quantities  is  the  term  for  the  process  of  deter¬ 
mining  the  probability  of  distributions  of  more  aggregate  (superior) 
quantities  given  the  distributions  for  less  aggregate  (subordinate) 
quantities  and  the  relationships  between  the  quantities.  The  process 
is  required  at  two  different  places  in  the  simulation  model.  First, 
it  is  used  after  information  is  received  from  an  action.  In  this  case, 
the  characteristics  measured  by  the  action  must  be  identified.  Then 
the  characteristics  that  are  superior  (more  aggregate)  are  (re) deter¬ 
mined  using  the  new  distributions  for  the  characteristics  measured  and 
the  "old"  distributions  for  all  the  other  characteristics.  The  second 
place  requiring  updating  was  mentioned  above — when  determining  supe¬ 
rior  preposteriors.  The  process  is  identical  in  both  cases. 

Updating,  or  aggregating,  can  be  accomplished  using  either  Monte 
Carlo  methods  or  the  Propagation  of  Error.  These  techniques  and  meth¬ 
ods  for  choosing  between  them  are  discussed  in  Appendix  D.  The  details 
of  updating  through  the  hierarchy  of  components  for  the  example  used  in 
the  study  are  contained  in  Appendix  C. 
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TERMINATION  AND  OUTPUT 

If  the  action  selected  was  a  terminal  action — that  is,  deliver  the 
system — then  the  output  items  must  be  recorded.  The  items  selected  for 
output  in  this  study  are: 

o  The  sequence  of  actions  selected. 

o  The  characteristics  of  the  system  that  was  delivered, 
o  The  alternative  design  that  was  delivered, 
o  The  incentive  contract  payoff. 

o  The  time  and  cost  (not  including  payoff)  of  the  project. 

These  items  constitute  a  history  of  the  development  project.  The  com¬ 
puter  program  is  capable  of  generating  any  number  of  project  histories 
by  re-establishing  the  initial  conditions.  Figure  11  illustrates  the 
process  for  generating  an  individual  project  history. 
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IV.  DECISIONMAKING  POLICIES  CONSIDERED 


The  decisionmaking  policies  examined  in  this  research  are  organi¬ 
zation  oriented.  The  criteria  for  decisionmaking  used  in  the  policies 
are  the  system  performance  characteristics,  the  program  cost  and  time, 
and  the  incentive  fee.  These  criteria  are  of  primary  importance  to  the 
organization  but,  perhaps,  of  less  importance  to  the  individuals  within 
the  organization . 

During  the  course  of  this  research,  nine  different  decision  poli¬ 
cies  were  considered.  Five  of  these  policies  were  selected  for  study 
using  samples  of  51  runs  each.  The  characteristics  of  the  policies  are 
summarized  in  a  tdble  later  in  this  section. 

POLICY  1: _  MAXIMIZE  EXPECTED  PAYOFF 

Summary  of  Policy  Characteristics 

Policy  1  is  a  surrogate  for  the  usual  analysis  of  a  choice  between 

* 

taking  a  terminal  action  or  taking  an  uncertainty  reducing  action. 

The  usual  decisionmaking  analysis  uses  preposterior  probability  distribu¬ 
tions  and  expected  value  calculations.  It  proceeds  as  follows:  (1) 

Calculate  the  net  expected  value  for  a  terminal  action,  EV  ,  using  the 

f  t 

prior  distributions  of  possible  outcomes.  (2)  Calculate  the  net  ex¬ 
pected  value  for  an  uncertainty  reducing  action  followed  by  a  terminal 

action,  EV  ,  using  the  preposterior  distributions  of  possible  outcomes. 

3 

(3)  If  EV  is  greater  than  EV  ,  then  select  the  uncertainty  reducing 

3  U 

action,  otherwise  select  the  terminal  action.  An  equivalent  form  of 

this  rule  is  to  calculate  the  difference  EVI  «  EV  -  EV ,  and  select 

s  s  t 

the  uncertainty  reducing  action  if  the  difference  is  positive,  other¬ 
wise  select  the  terminal  action.  In  the  case  of  ri  different  uncer¬ 
tainty  reducing  actions,  the  quantities  EVI.  =  EV .  -  EV  ,  are  calcu- 

1'  "V  u 

lated  for  all  actions  i  =  1,  . ..,  n.  If  EVI .  is  positive  for  at  least 

* V 

one  action,  then  select  the  action  that  has  a  maximum  value  of  EVI y 
otherwise  select  the  terminal  action. 

* 

See  Schlaifer,  op.  cit.  Chap.  33. 

Details  of  this  calculation  vary  from  policy  to  policy.  Explana¬ 
tions  of  these  variations  are  given  throughout  this  section. 
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Policy  1  differs  from  the  "usual"  procedure  just  described  with 
respect  to  the  means  of  calculating  the  EV  quantities.  Policy  1  calcu¬ 
lates  the  net  expected  payoff  for  an  information  buying  action  followed 
by  a  minimum  sequence  of  actions  required  to  reach  a  terminal  state. 

The  preposterior  analysis  is  carried  out  for  the  characteristics  that 
will  be  measured  by  the  action  being  evaluated.  The  times  and  costs 
of  the  actions  in  the  minimum  sequence  are  included  in  the  calculation, 
but  the  information  to  be  gained  from  these  actions  is  omitted.  Denote 
the  quantities  obtained  using  chis  procedure  by  EV 1^,  i  =  1,  . ..,  n. 

There  is  another  difference  between  Policy  1  and  the  "usual"  pro¬ 
cedure.  The  eligibility  rule  requires  that  certain  actions  be  executed 
before  a  terminal  action  becomes  eligible  for  selection.  When  the  pre¬ 
requisites  are  all  satisfied,  the  expected  payoff  of  the  eligible  ter¬ 
minal  action,  EV, ,  is  included  with  the  EV 1.  values  and  the  action 
having  the  maximum  value  is  selected.  The  same  result  is  obtained  by 

selecting  the  action  that  maximizes  the  quantity  EV.T1  .  =  EV  1.  -  EV , 

Z  z  z> 

if  EVT1  is  positive  for  at  least  one  action,  and  selecting  the  terminal 
action  otherwise.  This  is  the  same  as  the  "usual"  procedure.  The  dif¬ 
ference  occurs  when  the  prerequisites  are  not  satisfied  and  a  terminal 
action  is  not  eligible.  In  this  case,  the  action  that  maximizes  EV 1 
is  selected,  hut  EV  does  not  enter  into  the  selection.  However,  there 
is  still  a  strong  similarity  between  the  procedure  of  Policy  1  in  this 

case  and  the  "usual"  procedure.  There  is  a  value  for  EV  at  each  de- 

u 

cision  point  whether  or  not  a  terminal  action  is  eligible.  Denote  the 
value  of  EV ^  at  a  decision  point  where  a  terminal  action  is  not  eligible 
by  EV*.  Selecting  the  action  that  maximizes  EVl^  at  such  a  decision 
point  is  equivalent  to  selecting  the  action  that  maximizes  EV 1 .  -  EV * 
regardless  of  the  sign  of  the  difference. 

Of  all  the  policies  considered  in  this  research,  Policy  1  is  about 
the  closest  approximation  to  the  "usual"  procedure.  Consequently,  it 
is  labeled  "maximize  expected  payoff." 

Eligibility  Rule 


A  minimum  of  one  action  must  be  selected  from  each  of  the  ten  ac¬ 
tivity  categories.  An  assemble  and  test  must  be  done  before  a  deliver 
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may  be  selected,  and  one  action  from  each  of  the  eight  experiment  ac¬ 
tivities  must  be  done  before  an  assemble  and  test  may  be  selected. 
Hence,  a  project  history  will  contain  at  least  one  action  from  each  of 
the  ten  categories  shown  in  Table  2.  This  rule  is  reasonable  because 
each  of  the  activities  generates  certain  output  that  is  necessary  for 
fabrication  and  assembly  of  the  system.  For  example,  the  wind  tunnel 
tests  will  determine  the  shape  of  the  exterior  of  the  aircraft  and  the 
stress  analysis  creates  the  drawings  of  how  the  internal  and  external 
structures  will  be  put  together.  The  rule  is  not  reasonable  in  that 
it  does  not  require  that  each  of  the  ten  activities  be  directed  at  the 
same  system  design. 

Forecasting  Time  and  Cost 

Time  and  cost  forecasts  are  based  on  action  forecasts.  The  action 
forecast  is  based  on  the  eligibility  rule.  The  basic  forecast  is  that 
one  action  will  be  selected  from  each  activity  category  that  has  not 
yet  had  an  action  selected  from  it.  Because  of  the  mechanism  for  keep¬ 
ing  track  of  the  eligibility  requirements,  there  are  two  variations  in 
the  details  of  the  calculations.  In  the  first  variation,  the  action 
being  evaluated  will  be  the  first  chosen  from  its  category — the  eligi¬ 
bility  consideration  for  this  has  not  been  previously  satisfied.  In 
this  case,  the  basic  forecast  already  includes  the  action  being  evalu¬ 
ated.  In  the  second  variation,  the  action  being  evaluated  will  not 
be  the  first  chosen  from  its  category — the  eligibility  consideration 
for  this  category  has  been  previously  satisfied.  In  this  case,  the 
basic  forecast  does  not  include  the  action  being  evaluated.  The  two 
cases  can  be  expressed  as 


TA  =  TP  +  TU 


CA  =  CP  +  CU 


Case  I 


(7) 


Case  II 


(8) 


where  TA  is  the  forecast  time,  CA  is  the  forecast  cost,  TP  is  the  ac¬ 
cumulated  time  of  all  actions  that  have  been  selected,  CP  is  the  accu¬ 
mulated  cost  of  all  actions  that  have  been  selected,  TU  is  the  sum  of 
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the  times  of  one  action  from  each  activity  category  from  which  no  ac¬ 
tion  has  yet  been  selected,  &U  is  the  sum  of  the  costs  of  one  action 
from  each  activity  category  from  which  no  action  has  yet  been  selected, 
TE  is  the  time  of  the  action  being  evaluated,  and  CE  is  the  cost  of  the 
action  being  evaluated. 


Forecasting  Performance 

The  evaluation  rule  described  below  uses  probability  distributions 
for  the  two  system  performance  characteristics  that  appear  in  the  mul¬ 
tiple  incentive  contract.  To  obtain  a  forecast  of  these  distributions, 
the  characteristics  that  are  measured  by  the  action  to  be  evaluated 
must  be  determined  first.  Not,  using  the  precision  of  the  action,  the 
preposterior  distributions  for  the  characteristics  measured  are  estab¬ 
lished.  Then,  the  propagation  of  error  technique  is  used  to  determine 
the  preposterior  distributions  for  the  system  characteristics.  The 

preposterior  distribution  for  characteristic  c.  measured  by  action  a. 

*  ^  J 

is  determined  from 


E(ECo.))  =  E  Ac.) , 

pp  V  p  V 


a(E  Ac.))  =  a  Ac.) 
pp  v  p  t 


£L 


cr2(c.)  +  cx2.(c. ) 
p  i  J  t 


(9) 


(10) 


where  E(  )  indicates  the  mean  of  the  argument,  o(  )  indicates  the 
standard  deviation  of  the  argument,  the  subscript  pp  identifies  a  pre¬ 
posterior  quantity,  the  subscript  p  identifies  a  prior  quantity,  and 
the  subscript  j  identifies  a  sample  quantity — e.g.,  a  quantity  deter¬ 
mined  by  an  information  buying  action.  The  precision  of  each  activity 
is  expressed  by  the  coefficient  of  variation;  hence,  the  sample  stand¬ 
ard  deviation  equals  the  coefficient  of  variation  times  the  mean.  Con¬ 
sider  the  wind  tunnel  test  for  the  fuselage  and  tail,  action  number  42  as 


These  expressions  are  for  the  case  of  a  normal  prior  and  sampling 
distributions  with  known  sampling  variance.  See  Schlaiffer,  op.  cit., 
Chap.  34. 


shown  in  Table  2.  The  characteristic  measured  is  the  drag  coefficient 


for  the  fuselage  and  tail,  C 


for  C 


DfF' 


Suppose  that  the  prior  distribution 


DfF 


has  a  mean  of  .008  and  a  standard  deviation  of  .0008.  The 


precision  of  the  activity  is  shown  in  Table  2  to  be  3  percent.  Hence, 
the  sample  standard  deviation  is 


o’ .  0  ( (7n  J  =  .03  X  .008  =  .00024 , 
42  DjF 


The  preposterior  distribution  is  thus 


(ID 


V”  =  VCY}  =  *°08- 


a(fipp(cY))  =  -0008 


( .0008)" 


(.0008)  +  (.00024) 


-  \ 

2 


(12) 


.00076. 


(13) 


The  preposterior  distributions  for  the  system  performance  charac¬ 
teristics  are  determined  using  the  prior  distributions  for  the  compo¬ 
nent  characteristics  that  are  not  measured  and  the  preposterior  distri¬ 
butions  for  the  component  characteristics  that  are  measured.  For  the 
example  above,  the  preposterior  for  the  gross  take-off  weight  is  the 
same  as  the  prior  because  the  action  being  evaluated  does  not  produce 
information  about  weight. 

To  demonstrate  how  the  propagation  of  error  technique  is  used  to 
determine  the  preposterior  for  the  nautical  miles  per  pound  of  the  air¬ 
craft,  consider  thfe  following  case.  The  action  being  evaluated  is  ac¬ 
tion  number  42,  as  above.  The  system  design  for  which  the  preposterior 
is  to  be  determined  is  design  5^.  As  shown  in  Fig.  7,  this  system  de¬ 
sign  consists  of  engine  design  E  and  airframe  design  A  ;  and  aircraft 

ri  n 

design  A ^  consists  of  wing  design  W ^  and  fuselage  design  F  .  Suppose 

that  the  present  state  of  knowledge  for  the  drag  coefficient  of  wing 

design  WA  is  E(Cn  .  )  =  .016,  a(Cn  )  =  .00016,  E(e  )  =  .9,  and 
A  DjV>a  DfA  WA 

o(e  ,  )  =  .09.  Then,  using  Eqs.  (88),  (89),  (91)  and  (92),  the  prepos- 

WA 

teriors  for  the  aerodynamic  characteristics  of  the  airframe  are  given  by 


-45- 


>  Ya 


ppk  Y* 


pp 


pp 


+  .008  -  .024, 

(14) 

(.00076)2]7  -  .001771, 

(15) 

,<V>  =  -9* 

(16) 

A 

(e,  )  =  .09. 

(17) 

The  preposterior  for  the  nautical  miles  per  pound  is  given  by  E^  (NMI/LB) 
and  <Jpp(NMI/LB)  j  determined  as  indicated  in  Appendix  B. 


Evaluation  Rule 

The  value  assigned  to  any  action  is  determined  by  calculating  the 
expected  payoff  under  the  multiple  incentive  contract.  If  the  action 
has  as  its  object  some  component  that  may  be  used  with  a  number  of  dif¬ 
ferent  system  designs,  then  the  payoff  must  be  determined  for  each  de¬ 
sign  that  uses  the  component.  The  highest  value  is  selected  as  the 
value  of  the  action. 

There  are  four  components  to  the  expected  payoff  calculations: 

time,  cost,  and  the  two  performance  incentives.  The  time  and  cost  pay- 

* 

off  are  not  subject  to  uncertainty  in  the  present  formulation.  The 
time  and  cost  payoffs  are  determined  using  Eqs.  (2)  and  (3)  with  CA 
and  TA  given  by  Eqs.  (7)  or  (8). 

The  performance  payoffs  can  be  calculated  as  expectations  because 
the  probability  distributions  for  the  performance  values  are  available. 
The  expected  payoff  for  the  incentive  on  nautical  miles  per  pound  is 
given  by 

a  /  PSRNMI  -  E  \2 

EPANMI  =  -“r  ( RPSNMI  -  CPSNMI)  exp - 2E 

^  \  °pp  ) 

Some  ideas  for  changing  this  are  presented  in  Sec.  VII. 
Derivations  of  these  two  expressions  are  contained  in  Appendix  B. 
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+  CPSNMI  [  +  E  ~  PSRNMI 

\  /l^  PP 


PSRNMI  ~  E  \ 

Wo  ]’  <18> 

PP  ' 

where  a  denotes  cr  {NMI/LB)  j  E  denotes  E  {NMI/LB)  j  and  ERFC  is 
PP  PP  PP  PP 

the  cumulative  error  function.  All  other  symbols  are  as  defined  in 

Sec.  II. 

The  expected  payoff  for  the  incentive  on  gross  take-off  weight  is 
given  by 


{RPSNMI  -  CPSNMI) {E  -  PSRNMI ) 

+  - ^ - 22 -  ERFC 


PSRGTO  -  E 

EPAGTO  =  ( CPSGTO  -  RPSGTO )  exp - 

^  \  °pp 


-  RPSGTO 


II  +  E  -  PSRGTO 
2  PP  ) 


( RPSGTO  -  CPSGTO) (PSRGTO  -  E  )  / PSRGTO  -  E 

+  - - - ERFC  I - EE 


/2  a 


PP 


(19) 


where  cr  denotes  a  (GTO  WT).  and  E  denotes  E  ( GTO  WT)y  all  other 

PP  PP  PP  PP 

symbols  are  as  above. 

The  expected  incentive  payoff  is  given  by  the  sum  of  the  four 
terms . 


Preliminary  Results 

The  results  obtained  with  this  policy  were  sufficiently  interest¬ 
ing  to  warrant  its  use  in  large-sample  runs.  More  important  is  the 
fact  that  this  policy  constitutes  the  nbase  case."  All  other  policies 
represent  some  kind  of  deviation  from  this  policy. 

It  is  not  necessary  to  read  the  rest  of  this  section  unless  a 
knowledge  of  the  details  of  all  the  policies  is  required.  Table  3, 
coupled  with  some  of  the  details  of  Policy  1,  should  be  sufficient 
to  understand  the  remainder  of  the  study. 
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Table  3 

SUMMARY  OF  DECISIONMAKING  POLICIES  CONSIDERED 


Hum 

Eligibility 

Forecasts /In format  Ion 

Evaluation 

Preliminary 

Time  end  Cost 

Performance 

Results 

1 

"Hulalia 

Expected 

Payoff 

One  action  from 
each  experiment  ac¬ 
tivity  must  preceds 

assemble  end  test 
which  must  precede 
da  liver. 

Present  plus  un¬ 
completed  prere- 
qulsltee  plue 
action  being  con¬ 
sidered  If  prere¬ 
quisite  satisfied. 

Prepost ar lor s  for 
cherecterl sties 
meeeurad,  priors 
for  ell  othere. 

Maximize  expected 
payoff  uelng  mul¬ 
tiple  Incentive 
contract  end 
foreceeta . 

Reasonable  outcomes. 
Some  search  beyond 
minima  action 
requirement • 

2 

— 

"Straight 

Expected 

Pe  yo  f  f  ’ 

Assemble  end  test 
must  precede 
de 1 1 ver. 

Present  plus  as¬ 
semble  end  test 
plus  deliver. 

Same  ee  #1. 

Same  • e  1 . 

All  aequencee  of 
ectlone:  ■eeemble 

end  test  then 
deliver. 

3 

“Expected 

Payoff  Divided 
by  Coat1' 

Seme  as  *2. 

1 

Same  ee  *2. 

\  I 

i  _  —  •  — 

Same  ee  #1. 

1 

L  -- 

Maximlre  the  ratio 
of  expected  payoff 
(na  In  M)  divided 
by  the  cost  of  the 
action  being 
evaluated . 

If  values  negative 
when  first  assemble 
Is  selected,  then 
ell  eumblts 
•elected  before 
delivery 

4 

“Current 

Mean* 

Analyst  •' 

Sam*-  as  d  1 , 

Seme  as  a  1 . 

1 

. J 

r 

Uee  meena  of 

prlore  only. 

i  .... 

Same  as  » 1 . 

Reasonable  outcomes. 
No  search  beyond 
required  ectlone 

5 

Low  Coet 

Actions 

First" 

All  weight  esti¬ 
mates  and  s<*>sll 
wind  tunnel  test* 
first,  then  same 
as  *1. 

Same  as  *1, 

Seme  ee  *1. 

S  emc  a ■  1 , 

Reasonable  outcomes 
Some  search  beyond 
ml  ntmm  act  Ion 
r»qul rement t 

"Initial 

Prior  Means, 

Opt  (jslst  lc" 

— 

Seme  *s  *  1 . 

Same  as  U. 

1 

I 

Prepoeter lore  for 
character  let  Ice 
meeeured,  meena  of 
beet  Initial  de- 
elgn  for  ell 
othere . 

sexto  ss  *1, 

Fx trams ly  long 
aequencee  of 
actions 

’Initial 

Prior  Maine, 
Pessimistic 

as  1, 

Seme  as  *  1 . 

Same  ee  #6  except: 
vor»t , 

Same  as  *  1 . 

same  ••  for  Pnllcv 
No.  b. 

0 

Probebl  1 1  ty 
of  Success 

1 

same  as  "  1  . 

, 

L 

Evaluation  le 
not  dependant  on 
time  or  cost. 

i 

t  ...  .  _  . .. 

Same  ee  *1. 

Select  action 
that  maximize* 
probability  of 
technical  eucceaa 
until  delivery  la 
eligible,  then 
deliver  regerdleee 
of  success 

Reasonable  outcomes 
Search  le  frequently 
longer  then  mlnleuv* 

•o  ■ a  with  Pol lc 1  as 
b  end  7. 

i 

9 

1 

No  Failures 
Delivered'' 

r 

as  *‘l, 

1 

ame  ss  «8. 

1 

!  ....... 

Same  e.  M. 

. . 

ee  *0  ex¬ 
cept:  do  not 

1  deliver  If  eyetem 
doee  not  meet  or 
rtxceed  require¬ 
ment  a . 

omplete  failure 
le  frequent, 

POLICY  2:  STRAIGHT  EXPECTED  PAYOFF 


Summary  of  Policy  Characteristics 

Policy  2  yields  measures  that  are  surrogates  for  the  incremental 
value  of  an  uncertainty  reducing  action  except  that  no  experiment  cate¬ 
gory  actions  are  required  as  prerequisites  to  the  assemble  and  test 
category  actions.  The  only  difference  in  the  evaluation  procedure  is 
that  the  time  and  cost  forecasts  are  based  on  a  much  shorter  minimum 
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sequence  of  actions  required  to  reach  a  terminal  state.  Policy  2  is 
labeled  straight  expected  payoff  because  the  calculation  is  the  same 
as  for  Policy  1,  but  the  minimum  sequence  is  much  shorter. 

Eligibility  Rule 

The  same  as  for  Policy  1  except  that  none  of  the  experiments 
are  required. 


Forecasting  Time  and  Cost 

There  are  three  cases.  First,  if  the  action  being  evaluated  is 
a  deliver  action,  then  the  forecasts  are  given  by 


TA  *  TP  +  TE 
CA  =  CP  +  CE 


(20) 


Second,  if  the  action  being  evaluated  is  an  assemble  and  test  action 
then  the  forecasts  are  given  by 


TA  =  TP  +  TE  +  T(I)i 
CA  =  CP  +  CE  +  C(  I)K 


(21) 


where  T( I)  and  (7(1)  are  the  time  and  cost  for  a  deliver  action.  Third, 
if  the  action  being  evaluated  is  an  experiment  action,  then  the  fore¬ 
casts  are  given  by 


TA  -  TP  +  TE  +  T(I1)  +  T(I)l 
CA  -  CP  +  CE  4-  Ci II)  +  C(I)\, 


(22) 


f 


where  T(II)  and  (7(11)  are  the  time  and  cost  for  an  assemble  and  test 
action. 
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This  forecasting  scheme  is  based  on  the  following  action  forecast: 
Whatever  action  is  selected  at  the  present  decision  point,  the  subse¬ 
quent  actions  will  be  the  minimum  required  to  reach  a  terminal  state. 

Forecasting  Performance 

This  is  the  same  as  for  Policy  1. 

Evaluation  Rule 

This  is  the  same  as  for  Policy  1. 

Preliminary  Results 

All  test  sequences  for  this  policy  produced  the  same  results.  The 
first  action  selected  is  an  assemble  and  test  and  then  a  deliver.  This 
is  not  interesting  so  the  policy  was  not  used  in  large  sample  runs. 

POLICY  3:  EXPECTED  IAYOFF  DIVIDED  BY  COST 


Summary  of  Policy  Characteristics 

Policy  3  yields  a  measure  that  is  similar  to  a  rate  of  return. 

The  expected  payoff  calculations  using  Policy  2  are  divided  by  the  cost 
of  the  action  being  evaluated;  hence,  the  measure  is  the  amount  of  ex¬ 
pected  payoff  per  dollar  spent. 

Eligibility  Rule 

This  is  the  same  as  for  Policy  2. 

Forecasting  Time  and  Cost 

This  is  the  same  as  for  Policy  2. 

Forecasting  Performance 

This  is  the  same  as  for  Policy  1. 

Evaluation  Rule 

The  evaluation  rule  is  the  same  as  for  Policy  1,  except  that  the 
expected  payoff  is  divided  by  the  cost  of  the  action  being  evaluated,  C^. 
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Preliminary  Results 

Dividing  by  the  cost  of  the  action  being  evaluated  results  in 
high  values  for  low-cost  actions  and  low  values  for  high-cost  actions 
when  the  expected  payoff  is  positive.  However,  when  the  expected  pay¬ 
off  is  negative,  high-cost  actions  have  higher  (less  negative)  values 
than  low-cost  actions.  As  c.  consequence  of  the  former  condition,  all 
the  low-cost  actions  are  selected  at  the  beginning  of  the  project; 
this  is  not  undesirable,  but  a  much  simpler  policy  would  produce  the 
same  results.  The  latter  condition  is  disastrous.  Whenever  the  ex¬ 
pected  payoff  becomes  negative  then  all  high-cost  actions  will  be  se¬ 
lected  before  the  low-cost  actions.  In  the  hypothetical  project,  if 
the  expected  payoff  turns  negative  before  a  deliver  action  is  selected, 
all  assemble  and  test  actions  will  have  higher  values  than  the  deliver 
actions.  Consequently,  all  assemble  and  test  actions  will  be  selected 
before  a  deliver  action  is  selected.  This  happened  frequently  in  the 
preliminary  runs  with  this  policy  and  it  was  eliminated  from  further 
consideration. 

POLICY  A:  CURRENT  MEANS  ANALYSIS 

Summary  of  Policy  Characteristics 

Policy  A  is  intended  to  show  the  effect  of  using  a  single-valued 
estimate  of  the  technical  outcomes  as  the  basis  of  the  evaluation. 

The  incentive  payoff  determined  by  this  method  may  be  called  the  most 
likely  payoff  because  the  means  of  the  system  performance  distributions 
are  used  in  the  calculation  but  the  standard  deviations  are  not.  This 
type  of  decisionmaking  policy  might  be  used  in  an  organization  that 
did  not  believe  in  probability  distributions  over  possible  outcomes  or 
one  that  did  not  have  or  could  not  afford  an  information  processing 
system  that  could  perform  the  preposterior  analyses  and  expected  value 
calculations.  Policy  A  is  labeled  current  means  analysis. 

Eligibility  Rule 

This  is  the  same  as  for  Policy  1. 

Forecasting  Time  and  Cost 


This  is  the  same  as  for  Policy  1. 


-51- 


Forecasting  Performance 

This  policy  ignores  the  dispersion  in  the  distributions.  It  uses 
the  means  of  the  distributions  as  a  "best"  guess.  Hence,  the  preposte¬ 
rior  distributions  can  be  expressed  as 


E  {NMI/LB) 
PP 

-  Ep (NMI/LB) , 

(23) 

o  (NMI/LB) 
PP 

-  o, 

(24) 

E  ( GTO  WT) 
PP 

-  E  (GTO  WT), 

P 

(25) 

a pp (GTO  WT) 

-  0. 

(26) 

Evaluation  Rule 

The  evaluation  rule  is  the  same  as  for  Policy  1,  except  that  be¬ 
cause  the  forecast  standard  deviations  are  zero,  the  evaluation  expres¬ 
sions  reduce  to 


EPANMI 


(  RPSNMI (PSANMI  -  PSRNMI) ,  if  PSANMI  Z  PSRNMI 
\-CPSNMI (PSRNMI  -  PSANMI)  ,  if  PSANMI  <  PSRNMI 


(27) 


and 


EPAGTO 


RPSGTO (PSRGTO  -  PSAGTO) ,  if  PSRGTO  ^  PSAGTO 

[-CPSGTO  (PSAGTO  -  PSRGTO ),  if  PSRGTO  <  PSAGTO , 


(28) 


where  PSANMI  =  E  ( NMI/LB )  and  PSAGTO  =  E  ( GTO  WT) 
PP  PP 


Preliminary  Results 

All  project  histories  were  completed  in  the  minimum  number  of 
actions  required  by  the  eligibility  rule.  The  outcomes  were  reason¬ 
able,  and  the  policy  was  selected  for  large-sample  investigation. 


POLICY  5:  LOW-COST  ACTIONS  FIRST 


Summary  of  Policy  Characteristics 

Policy  5  is  intended  to  show  the  effect  of  examining  all  the  alter¬ 
native  designs  early  in  the  development  program.  This  corresponds 
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roughly  to  Rule  2  of  Klein,  et  al.  This  policy  differs  from  Policy  1 
only  in  the  eligibility  rule  that,  in  this  case,  requires  the  low-cost 
and  short-time  actions  for  the  four  wing  and  airframe  designs  to  be 
done  first.  Policy  5  is  labeled  low-cost  actions  first. 

Eligibility  Rule 

The  eligibility  rule  is  the  same  as  for  Policy  1,  except  that 
actions  25-32  and  41  of  Table  2  must  be  executed  first. 

Forecasting  Time  and  Cost 

This  is  the  same  as  for  Policy  1. 

Forecasting  Performance 

This  is  the  same  as  for  Policy  1. 

Evaluation  Rule 

This  is  the  same  as  for  Policy  1. 

Preliminary  Results 

The  preliminary  results  were  interesting.  This  policy  may  pro¬ 
duce  results  that  are  technically  superior  to  the  results  obtained 
using  Policy  1.  Selected  for  large-sample  investigation. 

POLICY  6:  INITIAL  PRIOR  MEANS,  OPTIMISTIC 

Summary  of  Policy  Characteristics 

Policy  6  is  intended  to  show  the  effect  of  not  using  current  in¬ 
formation  in  the  evaluation.  It  corresponds  to  an  organizational  situ¬ 
ation  in  which  the  engineering  group  that  is  evaluating  a  given  action, 
say  a  wind-tunnel  test  of  wing  design  u)^9  has  current  information  for 
the  characteristics  that  will  be  measured  by  the  given  Action,  in  this 
case  C  and  e  .  All  other  information  items  used  in  the  evaluation 

d?a  wa 

* 

*  Op.  cit.,  p.  4. 
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are  the  means  of  the  initial  probability  distributions  for  the  "best" 
design  at  the  beginning  of  the  project.  In  other  words,  an  engineer¬ 
ing  group  receives  current  information  about  the  characteristics  that 
will  be  measured  by  an  action  taken  by  the  group,  but  the  only  infor¬ 
mation  that  the  group  has  regarding  the  other  characteristics  is  the 
most  likely  estimate  for  the  best  design  at  the  beginning  of  the  proj¬ 
ect.  This  situation  corresponds  to  an  extreme  deviation  from  the 

* 

vice  of  Rule  4  of  Klein,  et  al.  The  calculation  procedure  is  the  same 
as  for  Policy  1  in  all  other  respects.  Policy  6  is  labeled  initial 
prior  means,  optimistic,  to  reflect  the  delayed  information  regarding 
the  best  design. 

Eligibility  Rule 

This  is  the  same  as  for  Policy  1. 

Forecasting  Time  and  Cost 

This  is  the  same  as  for  Policy  1. 

Forecasting  Performance 

For  the  characteristics  that  are  measured  by  the  action  being 
evaluated,  the  preposteriors  are  determined  using  Eqs.  (9)  and  (10)  as 
described  under  Policy  1  above.  For  all  the  other  component  character¬ 
istics,  the  means  of  the  preposterior  distributions  are  set  equal  to 
the  means  of  the  initial  probability  distributions  for  the  design  that 
yields  the  maximum  expected  performance  incentive  under  the  initial 
prior  distributions.  The  standard  deviations  of  the  other  component 
characteristics  are  set  equal  to  zero.  Then  the  propagation  of  uncer¬ 
tainty  is  used  to  determine  the  system  performance  preposteriors. 

This  policy  is  called  decentralized  because  when  a  given  action  is 
being  evaluated,  the  only  information  available  regarding  the  charac¬ 
teristics  of  the  other  components  is  the  initial  prior  means.  This 
is  contrasted  to  Policy  1  where  all  of  the  current  information  is  avail¬ 
able;  that  is,  there  is  a  centralized  source  of  all  information. 

-  - 
Ibid. ,  p.  5. 
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Evaluation  Rule 

This  is  the  same  as  for  Policy  1. 

Preliminary  Results 

The  sequences  of  actions  are  extremely  long.  This  results  in  long 
time,  high  costs,  and,  consequently,  low  incentive  pay-offs.  However, 
the  fact  that  so  much  search  for  a  technically  superior  system  was  taking 
place  was  sufficient  reason  for  running  large  samples  with  this  policy. 

POLICY  7:  INITIAL  PRIOR  MEANS,  PESSIMISTIC 

Summary  of  Policy  Characteristics 

Policy  7  is  a  minor  variation  of  Policy  6.  It  uses  the  means  of 
the  initial  probability  distributions  for  the  "worst”  design.  Accord¬ 
ingly,  Policy  7  is  labeled  "initial  prior  means,  pessimistic." 

Eligibility  Rule 

Tliis  is  the  same  as  for  Policy  1. 

Forecasting  Time  and  Cost 

This  is  the  same  as  for  Policy  1. 

Forecasting  Performance 

The  same  as  for  Policy  6  except  that  the  means  of  the  preposte¬ 
rior  distributions  for  the. component  characteristics  not  measured  by 
the  action  being  evaluated  are  set  equal  to  the  means  of  the  initial 
probability  distributions  for  the  design  that  yielded  the  minimum  ex¬ 
pected  performance  incentive  under  the  initial  prior  distributions. 

This  policy  is  called  pessimistic  while  Policy  6  is  called  optimis¬ 
tic  because  this  policy  uses  the  initial  "worst"  design  for  its  tech¬ 
nical  information  basis  while  Policy  6  uses  the  initial  "best"  design. 

Evaluation  Rule 


This  is  the  same  as  for  Policy  1. 
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Preliminary  Results 

A  sample  run  of  three  histories  under  this  policy  were  identical 

* 

to  the  first  three  sample  runs  under  Policy  6.  Hence,  this  policy 
was  not  selected  for  large-sample  investigation. 

POLICY  8:  PROBABILITY  OF  SUCCESS 


Summary  of  Policy  Characteristics 

Policy  8  ignores  the  time  and  cost  elements  in  making  decisions 
and  uses  only  the  probability  of  technical  success.  Such  a  policy 
might  be  used  when  the  contractor  wants  to  maximize  his  chances  of  re¬ 
ceiving  production  orders  and  he  suspects  that  such  orders  will  be  de¬ 
cided  on  the  basis  of  the  system  performance  characteristics.  Another 
situation  that  would  be  consistent  with  Policy  8  is  when  the  contrac¬ 
tor  wants  to  maintain  a  reputation  of  technical  success  and  does  not 
care  about  time  and  cost.  This  policy  is  labeled  probability  of  suc¬ 
cess.  It  could  also  be  called  minimize  technical  risk.  This  follows 
because  one  m\nus  the  probability  of  success  equals  the  probability 
of  failure. 

Eligibility  Rule 

This  is  the  same  as  for  Policy  1. 

Forecasting  Time  and  Cost 

The  evaluation  rule  for  this  policy  does  not  use  time  and  cost 
information. 

Forecasting  Performance 

This  is  the  same  as  for  Policy  1. 

Evaluation  Rule 

Select  the  action  that  maximizes  the  probability  of  technical  suc¬ 
cess.  Technical  success  means  that  both  system  performance  characteristics 

* 

The  same  initial  random  number  was  used  in  both  cases. 
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are  equal  or  superior  to  the  zero  performance  incentive  values,  PSRNMI 
and  PSRGTO.  As  defined  here,  the  two  performance  characteristics  are 
independent.  Hence,  the  probability  that  both  are  successful  is  equal 
to  the  product  of  the  probabilities  that  they  are  individually  success¬ 
ful,  or 


Pr(NMI / LB  >  PSRNMI ,  GTO  WT  ^  PSRGTO) 

=  Pr(NMI  LB  >  PSRNMI) Pv ( GTO  WT  £  PSRGTO ),  (29) 


where 


Pr(NMI  LB  >  PSRNMI)  =  \ ERFC 


PSRNMI  -  E  (NMI  LB) 
_ : _ PE _ 

/2  CTpp (NMI /LB) 


(30) 


and 


Pr (GTO  WT  <  PSRGTO) 


1  -  kERFC 


PSRGTO  -  E  ( GTO  WT) 
_ EE _ 

y/l  a  ( GTO  WT) 

PP 


(31) 


* 

and  all  symbols  are  as  defined  above. 

Selecting  the  action  that  maximizes  the  probability  of  technical 
success  is  pursued  until  a  deliver  action  becomes  eligible.  Then  that 
deliver  action  is  selected  regardless  of  whether  the  result  of  the  as¬ 
semble  and  test  yielded  a  technically  successful  system. 


Preliminary  Results 

The  outcomes  are  reasonable.  Sequences  of  actions  are  frequently 
longer  than  the  minimum  required  but  not  excessively  long  as  with  Pol¬ 
icies  6  and  7.  Policy  8  was  selected  for  large-sample  investigation. 

POLICY  9:  NO  FAILURES  DELIVERED 


Summary  of  Policy  Characteristics 

Policy  9  is  a  minor  variation  of  Policy  8.  It  attempts  to  elim¬ 
inate  the  inconsistency  in  Policy  8  of  selecting  actions  that  maximize 

* 

Derivations  of  these  expressions  are  presented  in  Appendix  B. 
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the  probability  of  technical  success  until  a  terminal  action  becomes 
eligible  and  then  selecting  that  deliver  action  whether  or  not  the  sys¬ 
tem  is  a  technical  success.  In  Policy  9,  the  System  that  will  be  de¬ 
livered  is  examined  to  make  sure  that  it  is  a  technical  success  before 
it  is  delivered.  If  it  is  not  successful,  then  the  development  contin¬ 
ues  for  the  remaining  designs  until  a  successful  system  is  achieved  or 
all  the  designs  are  exhausted.  Policy  9  is  labeled  no  failures  de¬ 
livered. 

Eligibility  Rule 

This  is  the  same  as  for  Policy  1. 

Forecasting  Time  and  Cost 

This  is  the  same  as  for  Policy  8. 

Forecasting  Performance 

This  is  the  same  as  for  Policy  1. 

Evaluation  Rule 

The  evaluation  rule  is  the  same  as  for  Policy  8,  except  that  when 
a  delivery  action  is  eligible,  the  system  to  be  delivered  is  examined 
to  see  if  it  is  technically  successful.  If  it  is,  then  it  is  delivered. 
If  it  is  not,  then  the  actions  that  have  not  yet  been  selected  or  pre¬ 
empted  are  picked  using  the  maximize  probability  rule  until  another 
delivery  is  eligible.  This  is  continued  until  a  technically  success¬ 
ful  system  is  obtained  or  all  (8)  systems  have  been  tried  with  none 
found  successful.  In  the  latter  circumstance,  the  program  writes  out 
the  message  ’’Attempted  all  alternatives . " 

Preliminary  Results 

Several  project  histories  failed  to  produce  a  technically  success¬ 
or 

ful  system.  Because  this  is  not  realistic,  the  policy  was  not  selected. 

* 

In  the  real  world,  repeated  failure  would  lead  to  a  redefinition 
of  technical  success.  To  make  this  policy  realistic  some  rule  for  this 
redefinition  would  have  to  be  incorporated.  Comments  on  this  are  pre¬ 
sented  in  Sec.  VII, 
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V.  RESULTS  FROM  THE  SIMULATION 


The  decisionmaking  policies  and  multiple  incentive  contracts  con¬ 
sidered  in  this  study  were  first  examined  using  preliminary  samples  of 
ten  runs.  Five  policies  and  two  contracts  were  selected  for  examina¬ 
tion  using  larger  samples.  The  sample  size  for  these  larger  runs  was 
determined  by  considering  the  statistical  significance  that  could  be 
achieved  in  relation  to  the  computer  time  required. 

Preliminary  runs  indicated  that  a  good  average  figure  for  the  time 
required  per  run  was  1  minute.  This  meant  that  ten  cases  (five  poli¬ 
cies  and  two  contracts)  would  require  10  hours  with  a  sample  size  of 
60.  Allowing  time  for  link-edit  operations,  50  appeared  to  be  a  reason¬ 
able  sample  size. 

The  means  and  standard  deviations  of  the  final  state  attributes 
for  some  of  the  preliminary  runs  (sample  size  of  20,  18  degrees  of  free¬ 
dom)  were  examined  to  determine  what  level  of  significance  could  be  ob¬ 
tained  with  a  two-tailed  t-test  comparison  of  the  means  for  a  sample 
size  of  51  (100  degrees  of  freedom),  assuming  that  the  same  results 
would  be  obtained.  The  findings  are  shown  in  Table  4.  The  first  col¬ 
umn  shows  the  approximate  t  values,  or  range  of  t  values,  obtained  from 
the  preliminary  runs.  The  second  column  shows  the  level  of  significance 

that  the  t  values  in  column  one  would  yield  if  the  sample  size  had  been 

* 

51  instead  of  10.  The  third  column  shows  the  level  of  significance 
that  corresponds  to  a  sample  of  101.  As  the  table  shows,  there  is  very 
little  to  be  gained  by  increasing  the  sample  size  above  51;  therefore, 
samples  of  51  runs  were  obtained. 

RESULTS 

The  outputs  for  each  project  history  include  the  final  state  and 
history  items  listed  in  Sec.  I.  All  output  was  obtained  on  punched 

_ 

The  percentage  values  in  the  table  were  obtained  by  linear  inter¬ 
polation  between  the  values  in  a  table  of  the  percentage  points  of  the 
t  distribution.  See  Appendix  Table  3  in  Albert  H.  Bowker  and  Gerald 
J.  Lieberman,  Engineering  Statistics ,  Prentice  Hall,  Englewood  Cliffs, 
N.J.,  1959. 
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Table  4 

ANALYSIS  OF  PRELIMINARY  t  VALUES  TO  DETERMINE  SAMPLE  SIZE 


Final 

State 

Attributes 

^ calc,  18 

a^ta,100 

1 

-  l^-Y  t 

\10f  ‘'calc,  18 

“1*0,200 

1 

I201)2  t 

\  10)  calc,  18 

Nautical  miles  per  pound 

1. 5-2.0 

<•0005-. 0005 

<•0005-.. . 

Gross  take-off  weight 

~.25 

~.288 

~.  216 

Payoff 

1.43 

.00087 

<•0005 

Cost 

2.0-2. 5 

<.0005-. . . 

<•0005-. . . 

Time 

2.0-2. 5 

<.0005-. . . 

<•0005-.. . 

cards  to  facilitate  future  analyses.  For  each  action  selected  in  any 
given  history  there  are  four  cards  that  specify  the  means  and  standard 
deviations  of  the  system  performance  characteristics.  If  N  actions 
are  selected  in  a  given  run,  then  there  will  be  4 N  cards  for  the  means 
and  standard  deviations.  There  are  two  cards  for  each  run  that  summa¬ 
rize  the  final  state  attributes  and  the  sequence  of  actions.  The  final 
state  attributes  and  the  sequence  of  actions  are  also  displayed  in  an 
easily  readable  format.  This  requires  5  cards  if  N  £  20  and  7  cards 
if  N  >  20.  Thus,  there  is  a  total  of  4/7  4-  7  cards  if  /V  ^  20  and  4/7  +  9 
cards  if  N  >  20.  A  total  of  459  (9  *  51)  project  histories  were  run. 
The  distribution  of  the  number  of  runs  of  various  lengths,  N9  is  shown 
below.  This  gives  a  total  of  30,606  cards. 

Run  length  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28 
No.  of  runs  220  15  5  4  1  2  91  9  6  3  1  7  7  12  9  17  21  14  15 

FINAL  STATES  OF  THE  DEVELOPMENT  PROCESS 

The  means  and  standard  deviations  for  the  five  attributes  of  the 
final  state  of  the  hypothetical  system  development  project  are  shown 
in  Table  5  for  the  five  policies  and  two  contracts  that  were  subjected 
to  large  samples. 
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Table  5 

MEANS  AND  STANDARD  DEVIATIONS  OF  FINAL 
STATE  ATTRIBUTES  BY  POLICY 


A inal  .  tat* 

Attmf  u ts* 

Policy  1 

Haximiss 
Fxp*ot*d  Payoff 

Polity  4 

Currant  Hsans 
Analysis 

Polity  S 

Lot,)  Cost 
Actions  firs  t 

Polity  6 
Initial  Prior 
Hsanst  C^timistio 

Polity  9 
Probability 
of  Suoosss 

Ms  ms 

Stcwtd  Dsv 

Hm  ms 

Stand  Dsv 

Hscns 

Stand  Dsv 

Hsans 

Stand  Dsv 

Hmcns 

Stand  Dsv 

Contract  A  low  lncaotlva 

Nautical  adlaa  par  poisid 

,095407+ 

.007734 

.091121+ 

.010090 

.096271+ 

.012613 

.0953*0+ 

.01*772 

.095328+ 

,006899 

Croaa  take  off  valght,  lb 

1266*0+ 

2*9  3 

126*96+ 

3*65 

128370+ 

292* 

128314+ 

3386 

129**3+ 

2607 

Payoff.  S  «  10  3 

21080+ 

9560 

20822+ 

10117 

16*95+ 

11556 

-  50  7  2  3a 

269  31 

9**7+ 

23280 

Coat,  5  *  10'3 

3047*8+ 

*65 

30*6  71+ 

0 

305166+ 

0 

321911+ 

4909 

306903+ 

3811 

Tla*.  vfca 

128+ 

6 

126+ 

0 

1*4+ 

0 

295+ 

*0 

1*9  ♦ 

*0 

-ontract  l»--hlgh  Incantlva 

Nautical  alia  a  par  pctnd 

.098215+ 

,006286 

.095724+ 

.0075*7 

.097283+ 

.008761 

.099768+ 

.01*089 

.095328+ 

.006*99 

Croaa  taka-off  valght,  lb 

128471+ 

2901 

128977+ 

3302 

128035+ 

3306 

128319+ 

2700 

129**3+ 

260  7 

Pavof f ,  S  «  10“3 

38921  + 

69591 

25  704+ 

6  59*2 

*1061+ 

7***9 

-38319+ 

666*9 

2230+ 

73979 

Coat  ,  $  *  I0'3 

30*801  + 

619 

30*6  71+ 

0 

30  59*1+ 

1716 

326  392+ 

2153 

306903+ 

3811 

Tla*.  «*a 

128+ 

6 

126+ 

0 

151  ♦ 

15 

325+ 

19 

1*9+ 

*0 

The  system  performance  outcomes  are  presented  in  scatter  diagram 
form  in  Fig.  12.  The  location  of  each  outcome  is  identified  by  a  num¬ 
ber  that  corresponds  to  the  system  design  (see  Fig.  7)  that  was  deliv¬ 
ered.  The  axes  of  the  diagrams  are  the  same  as  in  Fig.  8.  As  before, 
the  diagonal  lines  locate  the  zero  incentive  payoffs  for  system  perform¬ 
ance.  All  points  above  these  lines  correspond  to  losses  with  respect 

to  the  system  performance  incentive,  and  points  below  the  line  corres- 

* 

pond  to  gains.  The  further  the  points  are  away  from  the  line,  the  more 
negative,  or  positive,  is  the  amount. 

The  four  quadrants  determined  by  the  horizontal  and  vertical  dashed 
lines  are  the  same  as  in  Fig.  8.  Outcomes  in  Quadrant  IV  are  success¬ 
ful  with  respect  to  both  performance  characteristics;  outcomes  in  Quad¬ 
rant  II  are  unsuccessful  with  respect  to  both.  In  Quadrant  I,  outcomes 
are  successful  with  respect  to  fuel  economy  but  not  with  respect  to 
gross  take-off  weight;  in  Quadrant  III,  the  situation  is  reversed. 

* 

The  total  payoff  to  the  contractor  also  includes  a  fixed  fee  and 

the  incentives  with  respect  to  time  and  cost. 
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rt^.  72 — Scatter  diagrar-s  for  technical  out  cones 
(note  different  scale  for  12J) 
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The  crosses  in  the  figures  show  the  mean  plus  and  minus  the  stan¬ 
dard  deviation  of  the  technical  outcomes  for  each  contract  and  policy. 
All  the  crosses  are  combined  in  Fig.  12 j . 

Counting  the  system  designs  delivered  in  each  diagram  gives  the 
frequencies  of  delivery  shown  in  Table  6.  Design  5^  is  delivered  far 
more  frequently  than  any  other  design.  Figure  8  shows  why  this  is 
not  surprising;  design  S ^  is  initially  the  "best11  design. 

Counting  the  number  of  outcomes  in  each  quadrant  of  the  scatter 
diagrams  yields  the  frequencies  of  technical  success  and  failure  for 
each  contract  and  policy,  as  shown  in  Table  7. 

Histograms  for  the  five  attributes  are  shown  in  Fig.  13.  The  data 

for  these  histograms  were  generated  using  the  IBM  Scientific  Subroutine 

* 

TAB1 .  They  were  obtained  only  to  display  the  shape  of  the  distribu¬ 
tions  and  not  to  make  comparisons. 

SEQUENCE  OF  ACTIONS 

The  sequences  of  actions  generated  under  a  given  policy-contract 
combination  have  several  characteristics.  These  characteristics  can  be 
analyzed  in  various  ways  depending  on  the  available  information. 

Three  interesting  characteristics  that  can  be  examined  using  the 
data  generated  in  this  research  are:  Are  the  actions  chosen  in  an  or¬ 
der  that  is  uncertainty  reducing?  What  are  the  patterns  in  the  order 
of  selection?  What  are  the  reasons  for  sequences  containing  more  than 
the  minimum  number  of  actions  required  by  the  eligibility  rule? 

The  question  of  uncertainty  reduction  is  one  of  the  main  issues 
in  the  author’s  previously  referenced  work.^  Information  regarding 
change  in  uncertainty  over  time  that  was  generated  during  the  present 
research  will  be  analyzed  at  some  future  time. 

In  this  study,  only  the  patterns  in  the  order  of  selection  and  the 
reasons  for  more  than  the  minimum  number  of  actions  will  be  discussed. 
The  obvious  features  of  the  sequences  that  will  be  examined  include: 

* 

System/ Z60  Scientific  Subroutine  Package  (Z60A-CM-02X)  Version  III 
Programmer 'a  Manual ,  4th  ed. ,  H20-0205-3,  International  Business  Ma¬ 
chines  Corporation,  New  York,  1968. 

^Timson,  op.  cit.,  pp.  26-37. 
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Table  6 

FREQUENCIES  OF  DELIVERY  OF  ALTERNATIVE  SYSTEM  DESIGNS 


Policy 

Contract 

System  Design  Delivered 

S1 

55 

1 

A 

6 

— 

2 

— 

36 

-- 

7 

_ 

1 

B 

46 

I 

4 

— 

4 

A 

18 

— 

1 

— 

31 

— 

1 

— 

4 

B 

3 

1 

— 

— 

37 

6 

4 

— 

5 

A 

4 

1 

3 

3 

16 

12 

9 

3 

5 

B 

— 

— 

1 

— 

25 

12 

7 

6 

6 

A 

3 

1 

6 

2 

16 

11 

6 

6 

6 

B 

3 

— 

1 

— 

20 

14 

10 

3 

8 

- 

2 

2 

1 

— 

36 

4 

2 

3 

Table  7 

FREQUENCIES  OF  TECHNICAL  SUCCESSES  AND  FAILURES  BY  POLICY 

AND  CONTRACT 


Technical  Outcomes  1 
(Quadrants  I-IV) 

Policy  1 

Policy  4 

Policy  5 

Policy  6 

Policy  8 

A 

B 

A 

B 

A 

B 

A 

B 

Complete  success  (IV) 

31 

32 

20 

25 

32 

32 

26 

29 

32 

Partial  success  (I) 

12 

16 

12 

18 

7 

13 

11 

14 

11 

Partial  success  (III) 

1 

2 

17 

7 

7 

5 

10 

8 

4 

Complete  failure  (II) 

7 

1 

2 

1 

5 

1 

4 

— 

4 
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the  most  likely  order  of  the  first  eight  actions,  a  list* of  the  actions 
always  selected,  a  list  of  the  actions  never  selected,  the  minimum  num¬ 
ber  of  actions  required,  the  number  of  sequences  having  the  minimum 
number  of  required  actions,  the  average  number  of  actions  per  history, 
and  the  average  sequence  number  at  which  the  last  required  experiment 
is  executed.  These  items  are  shown  in  Table  8  for  each  policy  and 
contract. 

HISTORIES  OF  SYSTEM  PERFORMANCE  PROBABILITY  DISTRIBUTIONS 

The  means'  and  standard  deviations  of  the  system  performance  char¬ 
acteristics  at  each  decision  point  can  be  used  to  trace  the  changes 
over  time  in  the  probability  distributions  and  in  measures  such  as  the 
probability  of  technical  success.  These  "histories"  are  closely  re¬ 
lated  to  the  sequences  in  which  the  actions  are  selected  and  all  will 
be  analyzed  in  some  future  research. 


SUMMARY  CHARACTERISTICS  OF  SEQUENCES  OF  ACTIONS 
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VI.  POLICY  AND  CONTRACT  ANALYSIS  AND  COMPARISONS 


The  method  for  analyzing  decisionmaking  under  aggregate  uncertainty 
was  structured  to  be  used  for  the  evaluation  of  alternative  decision¬ 
making  policies  and  multiple  incentive  contracts.  To  that  end,  this 

* 

section  demonstrates  how  the  results  can  be  analyzed.  However,  it  is 
imperative  that  the  reader  keep  in  mind  that  conclusions  drawn  from 
these  analyses  are  valid  only  for  the  hypothetical  project  studied  here. 
Before  making  any  broad  generalizations,  it  is  necessary  to  examine 
many  projects  to  discover  the  conditions  that  favor  alternative  poli¬ 
cies  and  contracts. 

In  this  section,  the  results  described  in  Sec.  V  are  analyzed 
from  three  points  of  view.  First,  comparisons  are  made  between  the 
results  obtained  under  the  two  contracts  used  in  the  large  sample  runs. 
Second,  comparisons  are  made  between  the  results  obtained  for  the  five 
decisionmaking  policies  used  in  the  large  sample  runs.  Third,  the  re¬ 
sults  are  compared  to  some  of  the  earlier  studies. 

COMPARISONS  BETWEEN  CONTRACTS 

Comparisons  between  the  results  obtained  under  Contract  A  and  Con¬ 
tract  B  are  made  for  each  decisionmaking  policy.  Hence,  the  results 
of  Contract  A  and  Policy  1  are  compared  to  the  results  ot  Contract  B 
and  Policy  1,  Comparing  the  results  of  Contract  A  and  Policy  1  to  the 
results  of  Contract  B  and  any  other  policy  would  be  meaningless. 

The  comparisons  are  made  in  terms  of  the  final  state  attributes, 
the  frequencies  of  the  choices,  and  the  sequences  of  actions. 

Final  State  Attributes 

The  procedure  for  making  comparisons  using  the  final  state  attri¬ 
butes  is  to  first  test  for  a  significant  difference  between  the  variances 

* 

This  section  is  highly  analytical,  and  readers  who  are  not  inter¬ 
ested  in  extensive  F-  and  £-  ratio  analyses  can  skip  to  Sec.  VII  with¬ 
out  impairing  their  understanding  of  the  methodology  described  in  this 
study.  The  findings  of  this  section  are  summarized  in  Table  12  in  the 
next  section. 
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of  a  given  attribute.  This  is  done  using  a  two-tailed  F  test  at  the 

* 

5  percent  level  of  significance.  The  number  of  degrees  of  freedom  for 
all  cases  is  100.  Hence,  at  the  5  percent  level,  the  variances  are 
considered  to  be  aqual  if 


1.75  >  F  >  .571, 


(32) 


where  F  is  the  calculated  value  of  the  F  statistic. 

If  the  variances  are  not  significantly  different,  then  a  t  test 
at  the  5  percent  level  of  Significance  is  used  to  test  for  a  signifi¬ 
cant  difference  between  the  means.  In  these  cases  the  number  of  degrees 
of  freedom  is  100,  and  the  means  are  considered  to  be  equal  if 


t  >  t 


025,100 


1.984, 


(33) 


where  |t|  is  the  magnitude  of  the  calculated  value  of  the  t  statistic, 
and  t  q25  ^qq  value  of  t  at  the  2.5  percent  level  for  .100 

degrees  of  freedom. 

If  the  variances  are  significantly  different,  then  a  t  test  at 
the  5  percent  level  of  significance  is  used  to  test  for  a  significant 
difference  between  the  means.  In  these  cases  the  number  of  degrees  of 
freedom  must  be  calculated  and  a  different  value  of  the  t  statistic 
must  be  obtained  from  a  table  of  the  t  distribution.  As  above,  the 
means  are  considered  to  be  equal  if  the  magnitude  of  the  calculated  t 
is  greater  than  or  equal  to  the  table  value. 

The  results  of  the  comparisons  are  shown  in  Table  9.  Note  that 
the  means  of  the  payoffs  are  not  compared.  This  is  because  the  mean 
payoff  under  any  contract  can  be  adjusted  to  any  desired  level  by  chang¬ 
ing  the  fixed  fee  component  of  the  contract. 

The  comparisons  lead  to  the  following  general  conclusions: 

o  Technical  Performance .  Using  a  higher  rate  of  incentive  on 

performance  characteristics  (Contract  B)  appears  to  be  superior 

Discus s ion  of  these  tests  can  be  found  in  most  standard  statis¬ 
tics  texts  Such  as  Bowker,  op.  cit. 


CONTRACT  A  (LOW  INCENTIVE)  COMPARED  TO  CONTRACT  B  (HIGH  INCENTIVE)  USING  FINAL  STATE 
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in  terms  of  the  performance  values  obtained  than  using  a  low 
incentive  rate  (Contract  A).  Table  9  shows  that  the  mean 
fuel  economy  obtained  under  Contract  B  is  superior  to  that  ob¬ 
tained  under  Contract  A  for  all  policies.  This  result  is  sig¬ 
nificant  at  the  5  percent  level  for  two  of  the  policies.  The 
results  for  the  gross  take-off  weight  are  mixed  and  no  conclu¬ 
sion  can  be  drawn. 

o  Payoff .  The  higher  rate  of  incentive  on  performance  (Contract  B) 

produces  overwhelmingly  more  variance  on  payoff  than  does  a  lower 

* 

rate  (Contract  A). 

o  Time  and  Cost .  The  higher  rate  of  incentive  on  technical  per¬ 
formance  (Contract  B)  stimulates  more  search  than  a  lower  rate 
(Contract  A).  The  fact  that  times  and  costs  are  higher  means 
more  actions  were  taken;  hence,  there  was  more  search.  This 
result  is  very  significant  for  Policies  5  and  6.  The  compari¬ 
sons  for  Policies  1  and  4  do  not  support  this  distinction;  how¬ 
ever,  there  are  good  reasons  for  ignoring  these  inconsistencies, 
which  are  discussed  below. 

Looking  at  the  final  states  obtained  using  Policy  1,  it  is  ob¬ 
served  that  33  of  the  final  states  obtained  under  Contract  A  are  iden¬ 
tically  obtained  under  Contract  B.  In  addition,  3  more  final  states 
have  the  same  airframe,  and  another  has  the  same  airplane  efficiency 
factor.  This  leaves  only  14  cases  in  which  the  final  state  attributes 
are  totally  different.  Consequently ,  the  effective  sample  size  for 
comparisons  between  contracts  is  severely  reduced. 

The  reason  for  these  identical  runs  is  that  all  large-sample  runs 
were  started  with  the  same  random  number.  The  random  number  generator 
used  is  a  "pseudo  random  number  generator"  and,  consequently,  if  it 
is  called  the  same  number  of  times  on  two  separate  runs  that  begin 


The  evaluation  rule  for  Policy  8  is  independent  cf  the  contract; 
hence,  all  final  state  attributes  except  payoff  will  be  the  same  under 
any  contract.  Comparing  the  payoff  variances  for  Policy  8  gives  an  F 
value  of  .162  which  shows  a  significant  difference. 
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with  the  same  random  number,  then  the  final  random  number  will  be  the 
same  for  both  cases. 

Policy  A  also  has  several  identical  and  similar  runs  for  the  two 
* 

contracts.  However,  there  is  a  mechanical  reason  that  explains  this 
high  degree  of  similarity.  The  evaluation  rule  for  Policy  A  does  not 
consider  the  dispersion  of  possible  results.  It  uses  a  single  "best" 
estimate  of  the  outcome.  Consequently,  all  project  runs  using  Policy  A 
are  completed  in  the  minimum  number  of  required  actions.  Furthermore, 
the  actions  are  always  picked  in  descending  numerical  order.  This  is 
also  a  result  of  basing  all  evaluations  on  a  single-valued  estimate  of 
the  technical  results.  These  two  phenomena  result  in  the  pseudo  ran¬ 
dom  number  generator  being  called  the  same  number  of  times  for  every 
run,  and  the  actions  selected  at  each  decision  point  are  almost  always 
the  same. 

Regardless  of  the  reason  why  so  many  runs  in  a  sample  are  identi¬ 
cal,  the  fact  that  they  occur  is  *in  interesting  result.  It  means  that 
if  the  contractor  uses  either  Policy  1  or  Policy  A,  then  the  chances 
are  quite  high  that  identical  outcomes  will  be  obtained  regardless  of 
a  factor  of  ten  change  in  the  rate  of  performance  incentives.  However, 
in  spite  of  this  high  degree  of  identical  behavior,  the  fuel  economy 
obtained  under  the  high  incentive  contract  (Contract  B)  is  significantly 
superior  to  that  obtained  under  the  low  incentive  contract  (Contract  A). 

When  a  large  number  of  runs  turn  out  to  be  identical,  the  power  of 
the  statistical  analysis  can  be  improved  by  using  the  test  for  equality 
of  the  means  of  two  samples  when  the  observations  are  paired.  The  re¬ 
sults  of  this  test  are  shown  in  the  two  extreme  right  columns  in  Table  9. 
The  degrees  of  freedom  are  50  and  the  table  value  of  t  ^5  is  2.009. 

The  only  major  change  between  the  results  without  paired  observations 
and  the  new  results  is  that  for  Policy  A  the  difference  between  the 
gross  take-off  weights  is  significant  at  the  5  percent  level  using  the 
paired  observations  test  and  it  is  not  significant  using  the  t  test 
without  paired  observations. 

* 

There  are  32  identical  runs,  9  that  have  the  same  airframe,  and 
2  that  have  the  same  airplane  efficiency  factor. 

i 

This  test  is  described  in  Bowker,  op.  cit. ,  pp.  175-179. 
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The  choice  between  using  the  same  initial  random  number  for  all 
sample  runs  or  a  different  number  depends  on  what  is  important.  If  it 
is  more  important  that  the  similarities  or  differences  between  the  sam¬ 
ples  be  indicated  by  the  same  statistical  test,  then  different  initial 
random  numbers  should  be  used.  If  it  is  more  important  to  discover  a 
high  degree  of  identical  behavior,  then  the  same  initial  random  num¬ 
bers  should  be  used. 

Frequencies  of  Choices 

In  terms  of  the  frequencies  of  technical  success  and  failure,  the 
high  incentive  contract  (Contract  B)  results  in  more  technical  successes 

and  more  outcomes  that  are  satisfactory  with  regard  to  fuel  economy 

* 

and  are  overweigh  ;  than  the  low  incentive  contract  (Contract  A) .  In 
terms  of  the  frequencies  of  system  design  choices,  the  high  incentive 
contract  (Contract  B)  results  in  more  choices  of  system  designs  S ^  and 

than  does  the  low  incentive  contract  (Contract  A).  Also,  the  low 
o 

incentive  contract  (Contract  A)  results  in  more  choices  of  system  de¬ 
signs  S^9  S^y  and  S 

The  frequencies  of  design  choices  are  not  surprising  considering 
the  engine  used  in  the  various  designs  and  Fig.  8.  The  lines  of  con¬ 
stant  payoff  for  the  two  contracts  are  slightly  rotated  with  respect 
to  each  other  with  the  effect  that  the  low  incentive  contract  (Con¬ 
tract  A)  places  more  relative  incentive  on  gross  take-off  weight  than 
does  the  high  incentive  contract  (Contract  B) .  System  designs  S^9  S^9 
and  S ^  use  the  lower  weight  and  lower  fuel  economy  engine  (£^) ,  while 
designs  S and  S ^  use  the  higher  weight  and  higher  fuel  economy  engine 
(E^).  On  Fig.  8,  the  initial  probability  distributions  for  designs 
S and  S ^  are  farther  into  the  positive  payoff  region  of  the  high  in¬ 
centive  contract  (Contract  B)  than  for  the  low  incentive  contract  (Con¬ 
tract  A).  Also,  design  S is  farther  into  the  positive  payoff  region 
of  Contract  A  than  of  Contract  B  and  designs  S ^  and  S ^  are  not  as  far 
into  the  negative  payoff  region  of  Contract  A  than  of  Contract  B. 

* 

See  Table  7. 

See  Tab le  6 . 
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Sequence  of  Actions 

The  only  comparisons  that  can  be  made  using  the  sequences  of  ac¬ 
tions  are  concerned  with  the  amount  of  search.  The  general  impression 
given  by  the  items  in  Table  8  is  that  the  high  incentive  contract  (Con¬ 
tract  B)  generates  more  search  than  the  low  incentive  contract  (Con¬ 
tract  A).  More  search  would  be  expected  to  occur  when  a  larger  number 
of  actions  are  always  selected.  This  is  true  for  Policy  6  that  shows 
17  actions  always  selected  under  Contract  B  and  16  actions  always  se¬ 
lected  under  Contract  A. 

Similarly,  less  search  would  be  expected  when  more  actions  are  never 
selected.  This  is  indicated  by  Policy  1  (13  actions  under  Contract  A 
and  9  under  Contract  B  were  never  selected),  Policy  4  (13  actions  under 
Contract  A  and  11  under  Contract  B  were  never  selected),  and  Policy  5 
(2  actions  under  Contract  A  and  1  under  Contract  B  were  never  selected). 

More  obvious  indications  of  longer  search  are  the  average  number 
of  actions  per  simulated  project,  the  average  number  (serial)  action 
at  which  the  last  prerequisite  experiment  is  satisfied  and  the  average 
number  of  actions  beyond  the  minimum  required  by  the  eligibility  rule. 

All  three  of  these  indications  show  that  the  high  incentive  contract 
(Contract  B)  generates  longer  search  than  the  low  incentive  contract 
(Contract  A)  for  Policies  5  and  6.  Since  these  three  measures  are 
closely  related,  this  evidence  should  not  be  inferred  to  be  three  times 
as  strong  as  for  any  one  of  the  indicators. 

COMPARISONS  BETWEEN  POLICIES 

The  five  decisionmaking  policies  used  in  the  large-sample  runs 
were  formulated  with  specific  comparisons  in  mind.  The  eligibility 
rule,  forecasting  techniques,  and  evaluation  rule  used  in  Policy  1  are 
representative  of  the  kind  of  characteristics  that  are  desirable  in  a 
decisionmaking  policy.  Thus,  Policy  1  constitutes  a  proximate  criterion 
against  which  all  other  policies  are  compared. 

The  four  comparisons  are:  (1)  Policy  1  (complete  preposterior 
analysis  of  the  performance  probability  distributions)  compared  to 
Policy  4  (performance  represented  by  current  prior  means);  (2)  Policy  1 
(one  action  from  each  category)  compared  to  Policy  5  (all  weight  estimates 
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and  small  aircraft  wind  tunnel  tests  first,  then  one  each);  (3)  Policy  1 
(complete  preposterior  analysis  of  the  performance  probability  distri¬ 
butions)  compared  to  Policy  6  (preposterior  analysis  of  characteristics 
measured  but  initial  prior  means  for  all  others);  and  (4)  Policy  1 
(maximize  expected  payoff)  compared  to  Policy  8  (maximize  probability 
of  technical  success).  As  with  the  contract  comparisons,  it  is  meaning¬ 
less  to  compare  the  results  of  one  contract  and  policy,  say.  Contract  A 
and  Policy  1,  to  the  results  of  the  other  contract  and  any  other  policy, 
say,  Contract  B  and  Policy  4. 

The  comparisons  are  made  in  terms  of  the  final  state  attributes, 
the  frequencies  of  choices,  and  the  sequences  of  actions. 

Final  State  Attributes 

The  statistical  procedures  are  the  same  as  described  above.  The 
results  are  shown  in  Table  10. 

Policy  4  Compared  to  Policy  1.  Comparing  Policy  4  (current  prior 
means)  to  Policy  1  (preposteriors),  the  following  conclusions  may  be 
dr£  wn: 

1.  Technical  Performance .  Using  the  complete  preposterior  analy¬ 
sis  (Policy  1)  yields  superior  fuel  economy  as  compared  to 
using  only  the  prior  means.  This  result  is  significant  at  the 
5  percent  level  for  Contract  A.  The  gross  take-off  weight  re¬ 
sults  are  mixed. 

2.  Payoff .  The  comparisons  show  no  significant  difference. 

3.  Time  and  Cost .  Policy  1  exhibits  consistently  more  variable 
and  higher  times  and  costs.  All  of  these  results  are  signif¬ 
icant  at  the  5  percent  level  except  the  mean  costs. 

Policy  5  Compared  to  Policy  1.  Comparing  Policy  5  (all  low  cost 
actions  first)  to  Policy  1  (one  action  from  each  category)  ,  the  follow¬ 
ing  conclusions  may  be  drawn: 

1.  Technical  Performance.  The  fuel  economy  results  are  mixed  and 
not  significant.  The  gross  take-off  weight  results  are  con¬ 
sistent  but  still  not  significant. 
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2.  Payoff .  The  results  are  mixed.  The  one  significant  compari¬ 
son  (for  Contract  A)  shows  that  requiring  all  low  cost  actions 
to  be  done  first  (Policy  5)  reduces  the  payoff.  This  is  ex¬ 
pected  because  the  time  and  cost  incentives  are  the  same  for 
both  policies. 

3.  Time  and  Cost .  Requiring  the  low  cost  actions  to  be  done  first 
(Policy  5)  very  significantly  increases  the  mean  time  and  the 
mean  cost.  The  comparisons  of  the  variances  are  not  clear. 

For  the  low  incentive  contract  (Contract  A)  the  time  and  cost 
variances  are  significantly  less  for  Policy  5.  The  opposite 
is  true  for  the  high  incentive  contract  (Contract  B) . 

Policy  6  Compared  to  Policy  1.  Comparing  Policy  6  (initial  prior 
means)  to  Policy  1  (preposteriors) ,  the  following  conclusions  may  be 
drawn: 

1.  Technical  Performance .  Using  the  initial  prior  means  for  the 
initial  best  design  for  those  characteristics  not  measured 
(Policy  6)  instead  of  the  complete  distributions  (Policy  1) 
results  in  a  higher  variance  in  the  fuel  economies  for  both 
contracts.  The  gross  take-off  weight  shows  a  significantly 
higher  variance  for  the  low  incentive  contract.  The  means 
show  no  significant  difference. 

2.  Payoff .  The  complete  preposterior  analysis  (Policy  1)  is  far 
superior  to  the  initial  prior  means  analysis  (Policy  6)  for 
both  contracts.  The  variance  under  Policy  6  is  also  signifi¬ 
cantly  greater  than  under  Policy  1. 

3.  Time  and  Cost .  The  initial  prior  means  analysis  (Policy  6) 
produces  very  significantly  higher  means  and  variances  of  both 
time  and  cost  for  both  contracts. 

Policy  8  Compared  to  Policy  1.  Comparing  Policy  8  (probability 
of  success)  to  Policy  1  (expected  payoff) ,  the  following  conclusions 
may  be  drawn: 

1.  Technical  Performance .  The  results  are  independent  of  the 
contract.  Neither  performance  characteristic  comparison  is 
significant  at  the  5  percent  level. 
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2.  Payoff .  The  payoff  using  the  expected  payoff  rule  (Policy  1) 
is  significantly  superior  to  the  payoff  using  the  probability 
of  success  rule  (Policy  8)  for  both  contracts. 

3.  Time  and  Cost .  The  results  are  independent  of  the  contract. 

The  means  and  variances  of  both  the  time  and  the  cost  are  sig¬ 
nificantly  higher  using  Policy  8  than  using  Policy  1. 

Frequencies  of  Choices 

Comparisons  between  Policy  1  and  Policies  4,  5,  6 ,  and  8  in  terms 
of  the  frequencies  of  technical  successes  and  design  choices  are  sum¬ 
marized  in  Table  11.  The  general  conclusion  that  these  comparisons  lead 
to  is  that  Policy  1  is  as  good  or  better  than  the  other  policies  in 
terms  of  technical  successes — e.g.,  outcomes  in  Quadrant  IV  (see  Table  7). 
Also,  Policy  1  is  more  likely  to  result  in  outcomes  that  are  complete 
technical  failures — e.g.,  outcomes  in  Quadrant  II.  However,  the  abso¬ 
lute  number  of  technical  successes  is  several  times  the  number  of 


Table  11 

COMPARISONS  BETWEEN  POLICY  1  AND  POLICIES  4,  5,  6,  AND  8  IN  TERMS 
OF  FREQUENCIES  OF  TECHNICAL  SUCCESSES  AND  DESIGN  CHOICES 


I  ten 

More  Out cone e 
for  Policy  l 

Approximately 
Same  No.  of 
Outcomes  for 
Both  Policies 

More  Outcomes 
for  Policy  N 

Unclear 
as  to 
Dominance 

Policy  4 

Quadrant 

System  design 

II,  IV 

s3,  s5.  s7 

V  S8 

I,  III 
sv  s2,  S6 

— 

Policy  5 

Quadrant 

Svstern  design 

I.  11 

V  S5 

IV 

III 

s2»  s3*  V 

V  S  7  *  S8 

— 

Policy  6 

Quadrant 

System  design 

I,  11,  IV 

S5 

_ 

III 

V  V  S4 

Sl*  S7 

Policy  8 

Quadrant 

System  design 

I,  II 

V  V  S7 

IV 

S4*  S5 

V  s8 

III 

V  V  s8 

— 

NOTE:  The  assignment  of  the  results  into  one  of  the  four  quadrants  is 

according  to  their  relation  to  two  zero-incentive  performance  values — .088 
n  ml/lb  and  130,000  lb.  The  specific  quadrants  are  defined  as  follows*: 

The  aircraft  is  I:  inferior  in  weight,  superior  in  fuel  economy;  II:  in¬ 
ferior  in  regard  to  both  zero-incentive  values;  III:  inferior  in  fuel 
economy,  superior  in  weight;  and  IV:  superior  in  regard  to  both  zero- 
incentive  values. 
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complete  failures.  The  other  policies  are  more  likely  to  result  in 
outcomes  that  are  superior  with  regard  to  weight  but  inferioV  with  re¬ 
gard  to  fuel  economy — e.g.,  outcomes  in  Quadrant  III. 

With  regard  to  the  system  design  that  is  ultimately  delivered, 

Policy  1  is  more  likely  to  select  S,.,  while  the  other  policies  are  more 

likely  to  select  S_  and  S,.  There  appears  to  be  a  tendency  for  Pol- 
l  o 

icy  1  to  select  the  odd-numbered  designs  while  the  other  policies  tend 
to  select  the  even-numbered  designs.  The  odd-numbered  designs  have  the 
low  aspect  ratio  wing  and  the  even-numbered  designs  have  the  high  as¬ 
pect  ratio  wing.*  * 

Sequences  of  Actions 

As  with  the  comparisons  between  contracts  above,  the  Sequences  of 
actions  are  useful  only  to  analyze  the  amount  of  search.  The  general 
impression  given  by  the  information  in  Table  8  is  consistent  with  the 
statistical  tests  regarding  the  times  and  costs  as  reported  above.  Pol¬ 
icy  1  (complete  preposterior  analysis)  generates  longer  searches  than 
Policy  4  (current  prior  means  analysis),  but  the  difference  is  not  large. 
The  average  number  of  actions  for  Policy  1  is  10.25  under  Contract  A 
(low  incentives)  and  10.23  under  Contract  B  (high  incentives)  and  for 
Policy  4  it  is  10  for  both  contracts.  The  average  number  of  actions  for 
Policy  5  (all  low  cost  actions  first)  and  for  Policy  6  (initial  prior 
means  analysis)  are  overwhelmingly  greater  than  for  Policy  1.  The  Pol¬ 
icy  8  (probability  of  success  rule)  histories  are  moderately  longer. 

An  interesting  .difference  exhibited  by  Policy  8  is  that  it  is  the 
only  policy  that  evidences  search  after  the  minimum  requirements  of  the 
eligibility  rule  are  satisfied.  The  second  to  the  last  row  in  Table  8 
shows  the  average  number  of  actions  between  the  end  of  the  project  and 
the  action  that  satisfied  the  last  experiment  prerequisite.*  For  all 

~  * 

See  Table  7. 

+See  Fig.  7. 

Recall  that  an  assemble  and  test  action  cannot  be  selected  until 
a  minimum  of  one  action  has  been  selected  from  each  experiment  activ¬ 
ity  category. 
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policies  and  contracts,  except  Policy  8,  this  number  is  2.00.  For  Pol¬ 
icy  8,  it  is  2.21.  Two  actions  are  required  to  terminate  a  simulated 
project  after  the  last  prerequisite  experiment  is  completed — specific¬ 
ally,  one  assemble  and  test,  and  one  deliver.  Hence,  all  policies  ex¬ 
cept  Policy  8  exhibit  no  search  after  the  last  experiment  prerequisite 
is  satisfied.  The  2.21  figure  for  Policy  8  can  be  interpreted  that  21 
percent  of  the  time,  on  the  average,  one  additional  action  is  selected 
after  the  last  experiment  prerequisite  is  satisfied. 

COMPARISONS  WITH  EARLIER  STUDIES 

In  the  present  research,  the  final  state  of  the  system  develop¬ 
ment  project  is  characterized  by  the  five  attributes:  fuel  economy, 
gross  take-off  weight,  payoff,  program  cost,  and  time.  The  "goodness" 
of  a  decisionmaking  policy  can  be  judged  in  terms  of  these  attributes. 
It  is  important  to  note  that  there  are  two  points  of  view  for  judging — 
the  customer* s  and  the  contractor’s. 

Economic  Studies  and  the  Contractor’s  Point  of  View 

Most  of  the  earlier  studies  that  have  used  economic  analysis  of 

decisionmaking  have  asserted  that  from  the  contractor’s  point  of  view 

* 

optimal  policies  are  those  that  maximize  expected  paycff.  Consulting 
Table  5  shows  that  Policy  1  yields  the  highest  expected  payoff  with  one 
exception:  Policy  5  (all  low  cost  actions  first)  and  Contract  B  (high 

incentives).  Table  10  shows  that  the  comparison  is  significant  for 
Policies  6  and  8  but  not  for  Policies  4  and  5.  Hence,  the  exception 
can  be  ignored,  and  Policy  1  is  adjudged  superior  where  the  differences 
are  significant.  From  this  it  is  inferred  that  the  characteristics  of 
Policy  1  are  superior  to  the  characteristics  of  Policies  6  and  8.  This 
is  evidence  of  the  superiority  of  using  current  technical  information 
in  decisionmaking  as  opposed  to  old  information  (Policy  1  vs.  Policy  6) 
and  of  the  superiority  of  using  an  evaluation  rule  that  is  based  on  the 


T.  A.  Marschak  and  J.  A.  Yahav,  "The  Sequential  Selection  of 
Approaches  to  a  Task,"  Management  Sciences  ,  Vol.  12,  No.  9,  May  1966, 
pp.  627-647. 
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expected  payoff  as  opposed  to  one  that  is  based  on  only  the  technical 

* 

outcome  (Policy  1  vs.  Policy  8). 

The  Five  Rules  and  the  Customers  Point  of  View 

The  five  rules  of  Klein,  et  al.,  are  more  concerned  with  the  cus¬ 
tomer’s  point  of  view  than  with  the  contractors.  The  first  rule  re¬ 
lates  to  the  amount  of  detail  in  the  system  specification  and  is  beyond 
the  scope  of  the  present  study. 

The  second  rule,  which  says  that  preliminary  analysis  should  be 
made  on  a  number  of  alternatives,  is  represented  by  Policy  5  (all  low- 
cost  actions  first).  The  rule  implies  that  Policy  5  should  be  superior 
to  Policy  1  (one  action  from  each  category).  However,  Table  10  shows 
that  there  is  a  consistent  (i.e.,  for  both  contracts)  lack  of  differ¬ 
ence  between  the  two  policies  for  the  performance  characteristics  of 
the  system.  Furthermore,  Policy  5  is  consistently  different  for  time 
and  cost,  both  of  which  are  higher  because  of  the  additional  actions 
required  by  Policy  5.  However,  under  Contract  A  (low  incentives),  the 
payoffs  are  significantly  different  and  Table  5  shows  that  Policy  5 
is  superior  in  this  regard.* 

Rule  3  deals  with  the  timing  of  major  financial  commitment  to  a 
system  configuration.  None  of  the  policies  used  in  this  study  are  re¬ 
lated  to  this  rule. 

Policy  6  (initial  prior  means  analysis)  is  concerned  with  Rule  4, 
which  suggests  that  new  information  should  be  rapidly  transmitted  to 
technical  decisionmakers.  It  represents  extreme  departure  from  the  ad¬ 
vice  of  Rule  4,  while  Policy  1  (complete  preposterior  analysis)  repre¬ 
sents  strict  adherence.  Again,  there  is  a  consistent  lack  of  differ¬ 
ence  between  the  two  policies  for  the  system  performance  characteris¬ 
tics  and  there  is  a  consistent  difference  for  the  costs  and  times.  In 
this  case,  the  difference  is  in  accordance  with  the  rule.  Also,  the 

-  — 

Even  more  interesting  is  the  fact  that  for  the  project  used  in 
this  study,  the  rule  to  maximize  the  probability  of  technical  success 
did  not  produce  a  technically  superior  product. 

Op.  cit . ,  pp.  4-5. 

From  the  customer’s  point  of  view,  a  lower  payoff  is  superior 
because  a  payoff  is  money  out  of  his  pocket. 
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payoffs  are  consistently  different,  but,  from  the  customer’s  point  of 
view,  in  a  direction  that  makes  Policy  1  inferior* 

Rule  5  is  concerned  with  early  testing  of  components.  There  are 
no  early  test  actions  in  the  system  development  project  used  in  this 
study;  hence,  this  rule  is  not  examined. 
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VII.  CONCLUSIONS,  REFINEMENTS,  AND  EXTENSIONS 


CONCLUSIONS  ; 

This  research  was  undertaken  to  demonstrate  an  approach  to  the 
analysis  of  decisionmaking  in  system  development  that  combines  evalua¬ 
tion  at  the  system  and  program  level  with  actions  taken  at  lower  lev¬ 
els  in  the  work  breakdown  structure.  This  "linking-up ,  "  or  unification, 

is  made  possible  by  the  use  of  system  performance  design  equations  in 

* 

a  manner  previously  described  by  the  author.  The  demonstration  in¬ 
volved  the  development  and  use  of  a  computer  simulation  model  of  a  sys¬ 
tem  development  project. 

The  present  effort  highlights  two  important  areas  of  application. 
First,  the  computer  simulation  approach  appears  to  be  an  extremely  use¬ 
ful  method  for  analyzing  decisionmaking  policies  and  contracts  under  a 
variety  of  conditions.  The  present  model  demonstrates  the  capability, 
but  some  extensions  and  modifications  are  required  to  make  the  simulated 
project  more  realistic.  These  additions  include  such  items  as  more  de¬ 
tailed  system  test  actions.  Several  points  will  be  elaborated  below. 

The  second  area  of  application  is  risk  assessment.  The  hierarchy 
of  uncertainty  presents  probabilistic  information  regarding  the  tech- 
cal  outcomes  for  each  system  design.  This  information  can  be  used  for 
technical  risk  assessment.  Policy  8  (probability  of  success)  is  a  risk 
criterion.  The  probability  of  technical  success  is  equal  to  one  minus 
the  probability  of  technical  failure,  by  definition.  Furthermore,  para¬ 
metric  cost-estimating  techniques  make  it  possible  to  extend  the  pro¬ 
cedure  to  cost-risk  assessment.  This  is  commented  on  briefly  below. 

The  general  picture  is  that  Policy  1  (maximize  expected  payoff) 
is  superior  to  all  other  policies.  In  terms  of  the  differences  between 
the  elements  of  the  policies  this  implies  that 

o  Complete  preposterior  analysis  is  superior  to  analysis  based 
only  on  the  current  prior  means  (Policy  A),  and  it  is  also  su¬ 
perior  to  analysis  based  on  the  initial  prior  means  for  the 
characteristics  not  measured  (Policy  6). 

★ 

Timson,  op.  cit.,  pp.  5-18. 
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o  Requiring  that  all  low  cost  actions  be  done  first  (Policy  5) 
is  inferior  to  not  so  requiring, 
o  Evaluation  based  on  the  expected  payoff — i.e.,  incorporating 
time  and  cost  considerations — is  superior  to  evaluation  based 
on  the  probability  of  technical  success  (Policy  8)  alone. 

r 

It  is  important  to  note  that  the  evidence  for  the  superiority  of 
Policy  1  is  largely  concentrated  in  those  items  that  relate  to  the 
amount  of  search — i.e.,  time,  cost,  and  sequence  of  actions.  If  the 
contractor  can  persuade  the  customer  to  change  the  time  and  cost  incen¬ 
tive  depending  on  the  decision  policy,  then  there  could  be  changes  in 
the  results.  This  would  be  especially  true  of  Policy  5.  Under  this 
policy,  the  minimum  number  of  actions  taken  is  greater  than  for  all 
other  policies.  If  this  is  desired  by  the  customer,  then  he  should  in¬ 
crease  the  compensation  to  the  contractor  for  the  extra  effort. 

It  is  important  to  repeat  the  caveat  that  the  conclusions  are  valid 
only  for  the  conditions  of  the  hypothetical  project  used  in  this  study. 
Other  projects  with  other  conditions  might  yield  different  conclusions. 
An  overall  view  of  the  conclusions  that  may  be  drawn  from  the  compari¬ 
sons  between  the  policies  ate  summarized  in  Table  12.  For  each  pos¬ 
sible  comparison,  the  table  shows  the  number  of  the  superior  policy. 

If  the  comparisons  showed  mixed  results  for  the  two  contracts,  then 
"unclear"  is  entered.  If  the  results  are  not  significant  at  the  5  per¬ 
cent  level  or  better,  then  "insignificant"  is  entered. 

The  general  conclusions  that  may  be  drawn  from  the  comparisons 
between  Contract  A  (low  incentive)  and  Contract  B  (high  incentive)  are 
summarized  as  follows: 

o  Nautical  Miles  per  Pound  (NMI/LB) .  Contract  B  is  better  than 
Contract  A,  but  two  out  of  four  comparisons  are  insignificant, 
o  Gross  Take-off  Weight  (GTO  WT).  The  results  are  unclear. 

Three  out  of  four  comparisons  are  insignificant, 
o  Payoff .  Contract  B  has  a  higher  variance. 

o  Cost .  Contract  A  is  better  than  Contract  B,  but  two  out  of 
four  comparisons  are  insignificant. 

Time .  Contract  A  is  better  than  Contract  B,  but  two  out  of 
four  comparisons  are  insignificant. 
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o  Frequency  of  Success .  Contract  B  is  better  than  Contract  A. 
o  Sequences  of  Actions .  Contract  A  is  better  than  Contract  B. 

The  general  picture  given  by  the  conclusions  is  that  high  incen¬ 
tives  on  the  system  performance  characteristics  stimulate  more  search 

* 

for  a  technically  superior  product.  This  search  results  in  improved 
fuel  economy  performance  end  more  technical  successes  for  all  policies 
that  are  sensitive  to  incentives.  The  improvement  is  significant  at 
the  5  percent  level  for  two  of  the  four  policies.  However,  the  in¬ 
creased  search  means  higher  costs  and  longer  times  and  higher  uncer¬ 
tainty  regarding  the  incentive  payoff.  Whether  the  increased  techni¬ 
cal  success  is  worth  the  increased  time,  cost,  and  payoff  uncertainty 
is  a  question  that  is  beyond  the  present  study. 

The  final  state  attributes  are  not  all  equally  good  as  indicators 
of  the  differences  between  the  policies  and  contracts.  The  differen¬ 
ces  are  most  obviously  indicated  by  the  payoff,  time,  and  cost  attri¬ 
butes  of  the  final  states.  The  fact  that  the  system  performance  char¬ 
acteristics  did  not  exhibit  many  significant  comparisons  may  be  due  to 
the  particular  technology  used  in  this  study.  The  percentage  uncertainty 
regarding  the  system  performance  characteristics  is  less  than  the  per¬ 
centage  uncertainty  regarding  the  aerodynamic  characteristics  at  the 
component  level  (see  Fig.  7).  The  same  phenomenon  holds  for  the  gross 
take-off  weight.  In  the  weight  case,  it  is  due  to  the  large  fixed 
weight  taken  by  the  "other"  systems.  However,  for  the  fuel  economy 
case,  it  is  due  to  the  whole  set  of  design  equations. 

Regardless  of  the  reason  for  the  reduction  of  percentage  uncer¬ 
tainty  phenomenon,  it  means  that  for  given  variations  in  technical  out¬ 
comes  at  low  levels,  the  percentage  variations  at  higher  levels  will 
be  less.  In  view  of  this,  it  is  not  surprising  that  the  system  perform¬ 
ance  characteristics  are  not  strong  indicators  of  the  relative  superi¬ 
ority  of  policies  or  contracts.  Of  course,  this  is  merely  a  hypothesis. 


In  some  of  the  preliminary  runs,  it  was  observed  that  the  con¬ 
tract  with  five  times  the  rate  of  incentive  ( NEUT  *  5  in  Table  1)  did 
not  stimulate  more  search  than  the  low  incentive  contract.  Hence,  there 
may  be  a  threshold  phenomenon.  This  may  be  worthy  of  further  study. 
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To  examine  it  would  require  conducting  at  least  one  other  study  of  the 
same  nature  as  the  present  study,  but  with  a  technology  that*  exhibits 
increasing  uncertainty  as  one  proceeds  up  through  the  component-subsys¬ 
tem  hierarchy.  However,  if  the  hypothesis  is  true,  then  more  attention 
should  be  given  to  placing  incentives  on  low-level  technical  character¬ 
istics  in  situations  where  the  uncertainty  hierarchy  exhibits  decreas¬ 
ing  percentage  uncertainty.  This  is  one  of  the  alternatives  suggested 
in  the  discussion  of  extensions  below. 

REFINEMENTS 

There  are  several  features  of  the  model  of  the  system  development 
process  used  in  this  study  that  could  be  refined  to  make  the  model  more 
representative  of  the  real-world  process  of  development  *  Most  of  these 
refinements  require  significant  increases  in  the  size  of  the  computer 
program.  In  addition,  the  decisionmaking  policies  can  be  refined  in 
several  ways.  Six  of  the  more  obvious  refinements  are  outlined  in 
this  section. 

Including  Uncertainty  About  Times  and  Costs 

The  times  and  costs  of  the  various  actions  in  the  hypothetical 
project  are  constants.  This  is  unrealistic  in  two  respects.  First,  in 
certain  cases  the  times  and  costs  of  actions  will  vary  with  the  physi¬ 
cal  and  performance  characteristics  of  the  objects  of  the  actions.  For 
example,  it  seems  reasonable  that  all  other  things  being  equal,  it 
should  take  longer  to  fabricate  a  structural  member  that  weighs  more 
than  another.  Similarly,  the  equipment  and  machinery  used  in  the  fab¬ 
rication  would  be  more  expensive  because  it  would  have  to  be  more  sub¬ 
stantial.  This  type  of  variation  in  times  and  costs  could  be  incorpo¬ 
rated  by  expressing  the  times  and  costs  as  some  relationship  involving 
the  characteristics  of  the  components  or  subsystems.  Uncertainty  re¬ 
garding  the  times  and  costs  would  arise  from  the  uncertainty  regarding 
the  characteristics  of  the  components  of  subsystems. 

The  second  source  of  variation  in  times  and  costs  stems  from  the 
ability  to  vary  the  intensity  with  which  activities  are  carried  out. 
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By  varying  the  intensity  of  an  activity,  it  is  possible  to  make  trade¬ 
offs  between  time  and  cost.  In  terms  of  the  model  of  system  develop¬ 
ment  used  in  this  study,  such  variations  constitute  the  definition  of 
alternative  actions.  For  example,  the  time  for  a  stress  analysis  for 
a  wing  is  given  in  Table  2  as  16  weeks  and  the  cost  as  $500,000.  How¬ 
ever,  by  working  overtime  or  hiring  more  personnel  it  may  be  possible 
to  complete  the  activity  in  12  weeks  at  a  cost  of  $750,000.  Of  course, 
there  may  also  be  a  change  in  the  precision  (as  indicated  by  the  coef¬ 
ficient  of  variation)  of  the  results. 

Both  of  these  complications  would  require  improvements  in  the  pro¬ 
cedure  of  forecasting  times  and  costs. 

Sequential  Versus  Parallel  Choices 

In  actual  development  projects  many  activities  are  undertaken  at 
th i  same  time.  The  model  used  in  this  study  permits  only  one  activity 
at  a  time.  Changing  the  model  to  permit  more  than  one  activity  to  be 
undertaken  at  any  one  time  requires  several  additional  procedures. 

These  procedures  are  required  to  evaluate  and  store  the  values  obtained 
for  all  eligible  actions  at  each  decision  point.  Presently,  for  strictly 
sequential  selection,  only  the  value  of  the  highest  value  action  must 
be  retained.  Also,  there  must  be  a  means  of  determining  how  many  ac¬ 
tions  may  be  undertaken  at  any  one  time.  To  do  this,  the  resources 
available  must  be# specif ied .  To  make  the  action  selections,  a  scheme 
for  matching  the  resource  requirements  of  actions  with  the  available 
resources  must  be  devised  for  use  in  conjunction  with  the  schedule  of 
action  values.  The  additional  computer  memory  requirements  to  imple¬ 
ment  such  a  scheme  would  be  very  large. 

Expanding  the  Basis  of  Action  Evaluation  Beyond  the  Attributes  of  the 

Final  State  of  the  Development  Process 

In  many  situations  the  contractor  is  motivated  by  more  than  the 
attributes  of  the  final  state  of  the  development  process  as  defined 
in  this  work.  Specifically,  a  strong  motivating  factor  is  the  pos¬ 
sibility  of  production  orders  that  may  follow  the  development  project. 
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Whethar  or  not  there  is  a  follow-on  production  order  (and  its  size) 
depends  on  the  final  characteristics  of  the  system  when  development 
is  completed.  It  should  be  possible  to  include  some  forecast  of  pos¬ 
sible  production  orders  as  a  function  of  the  attributes  of  the  final 
state  of  development  and  thereby  extend  the  decisionmaking  time  hori¬ 
zon.  Of  course,  the  fact  that  the  present  model  does  not  include  such 
a  capability  should  not  detract  from  its  usefulness.  There  are  many 
development  situations  in  which  only  one  complete  system  is  required, 
or  at  most  a  few.  These  situations  are  quite  prevalent  in  the  space 
program,  where  only  one  satellite  of  a  given  type  is  required. 

Incorporating  Time  Lags  in  the  Information  Network 

Policy  6  was  constructed  to  demonstrate  the  effect  of  not  using 
all  the  current  technical  information  to  make  decisions.  The  situation 
corresponds  to  one  where  decisionmakers  have  current  information  re¬ 
garding  only  their  part  of  the  project.  Their  information  regarding 
the  other  parts  of  the  project  consists  of  the  initial  best  estimate. 

It  would  perhaps  be  more  realistic  to  assume  a  less  extreme  delay  in 
communication.  This  could  be  done  by  communicating  information  ob¬ 
tained  from  actions  that  precede  the  most  recent  action  but  not  from 
the  most  recent  action  itself.  It  might  be  useful  to  examine  the  im¬ 
plications  of  delays  of  various  lengths. 

Revision  of  the  Definition  of  Technical  Success 

Policy  9  was  not  satisfactory  because  complete  failure  was  fre¬ 
quent.  This  could  be  remedied  by  revising  the  definition  of  technical 
success  each  time  an  assemble  and  test  did  not  succeed.  This  would  be 
more  consistent  with  real-world  experience.  When  a  development  project 
does  not  yield  the  desired  results,  the  contract  is  sometimes  renego¬ 
tiated,  or  it  may  be  cancelled  if  the  results  are  sufficiently  bad. 

This  could  easily  be  incorporated  in  the  model  used  in  the  present  study. 
All  that  is  needed  is  a  subroutine  to  redefine  the  contract  in  the  event 
of  failure.  If  the  cancellation  feature  is  desired,  this  could  be  in¬ 
corporated  in  the  same  routine.  Cancellation  should  be  based  on  the 
results  expected  from  the  designs  not  yet  tested  rather  than  the  design 
that  just  failed. 
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Catastrophes  and  Breakthroughs 

A  very  important  aspect  of  system  development  in  the  real  world 
is  the  occurrence  of  technological  breakthroughs  and  catastrophes.  These 
extremes  can  be  incorporated  in  the  model  used  in  the  present  research 
by  having  four  probability  distributions.  Three  of  the  distributions 
would  correspond  to  a  breakthrough,  a  catastrophe,  and  a  "normal"  out¬ 
come.  The  fourth  distribution  has  three  outcomes:  one  would  correspond 
to  a  breakthrough,  another  to  a  catastrophe,  and  a  third  to  a  normal 
outcome.  The  probability  of  the  extreme  outcomes  would  be  very  low, 
say,  1  chance  in  100,  and  the  normal  outcome  would  have  a  high  probabil¬ 
ity  of  occurrence,  say,  98  chances  in  100.  Each  time  an  action  that  may 
lead  to  a  breakthrough  or  catastrophe  is  executed,  the  fourth  distribu¬ 
tion  would  be  sampled.  Then,  depending  on  the  outcome,  one  of  three 
other  distributions  would  be  sampled  to  determine  the  value  of  the  out¬ 
come.  This  procedure  would  be  useful  only  if  there  was  some  basis  for 
anticipating  catastrophes  and  breakthroughs . for  given  actions. 

EXTENSIONS 

There  are  many  ways  in  which  the  present  work  can  be  extended. 

The  most  obvious  extension  is  to  similar  studies  involving  other  tech¬ 
nologies  or  different  decisionmaking  policies.  One  interesting  policy 
variation  is  the  "compound"  policy.  A  compound  policy  consists  of  sev¬ 
eral  simple  policies,  such  as  those  examined  in  this  study,  that  are 

v 

used  for  decisionmaking  at  different  times  during  a  development  proj¬ 
ect.  It  seems  logical  that  the  importance  of  the  various  aspects  of 
the  system  and  the  project  may  vary  as  the  program  progresses.  Hence, 
it  might  be  advantageous  to  vary  the  decisionmaking  policy  used  when 
there  is  a  change  in  the  conditions  under  which  the  program  is  being 
conducted.  An  example  of  such  changes  in  conditions  is  a  technological 
catastrophe  or  breakthrough. 

A  great  deal  of  additional  work  can  be  done  in  the  area  of  multiple 
incentive  contracts  using  the  methods  developed  in  this  research.  The 
inclusion  of  two  contracts  in  the  present  study  was  only  intended  to 
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demons  trate  the  usefulness  of  the  methods  for  such  analysis.  Some 
of  the  interesting  aspects  of  incentive  contracting  that  can  be  stud¬ 
ied  include  the  effect  of  placing  incentives  at  different  levels  of 
the  technical  hierarchy,  e.g.,  placing  incentives  on  c  instead  of 

mu  lb.  -  * 

Another  variation  is  the  effect  of  placing  incentives  on  the  times 

of  various  events  during  development — e.g.,  first  static  test,  or  first 

flight  test — as  opposed  to  having  only  one  incentive  on  the  final  de- 
* 

livery  date. 

Perhaps  the  most  interesting  and  valuable  extension  of  the  present 
work  is  related  to  the  question  of  allocation  of  resources  between  re¬ 
search  activities  and  development  activities.  The  distinction  between 
the  two  aspects  of  the  state  of  knowledge  in  system  development  makes 
this  extension  possible.  The  design  equations  are  almost  never  known 
with  precise  certainty.  In  many  cases,  the  amount  of  uncertainty  re¬ 
garding  these  equations  is  sufficiently  small  that  it  can  be  safely  ne¬ 
glected.  However,  in  many  other  cases  the  uncertainty  regarding  the 
equations  cannot,  or  at  least  should  not,  be  ignored.  The  uncertainty 
can  take  two  forms:  (1)  regarding  the  numerical  constants  in  the  equa¬ 
tions — the  coefficients  and  exponents;  and  (2)  regarding  the  analytical 
form  of  the  equations  themselves.  Using  the  methods  described  in  this 
study,  it  should  be  possible  to  analyze  whether  uncertainty  about  the 
component  characteristics  or  uncertainty  about  the  relationships  be¬ 
tween  the  characteristics  contribute  more  to  the  total  uncertainty  re¬ 
garding  the  ultimate  performance  of  the  system.  This  information  could 
be  used  as  the  basis  for  allocating  resources  to  acquiring  more  infor¬ 
mation  about  the  components — development — or  allocating  resources  toward 
improving  the  knowledge  regarding  the  design  equations — research. 

An  extension  of  the  present  work  that  has  a  more  immediate  appli¬ 
cation  is  the  area  of  cost  estimating.  A  cost-estimating  relationship 
is  a  functional  relationship  between  the  characteristics  of  a  system 
or  piece  of  equipment  and  its  cost.  Hence,  the  methods  developed  in 

_ 

Rule  1  of  Klein,  et  al.,  implies  that  the  fewer  the  restrictions, 
the  better  the  expected  results.  However,  multiple  time  incentives  are 
common  practice. 
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this  study  can  be  used  to  generate  probabilistic  cost  estimates  during 
the  course  of  development  of  a  system  or  piece  of  equipment. 

Anott er  interesting  area  in  which  the  present  work  might  be  use¬ 
ful  is  in  teaching  classes  on  policies  for  decisionmaking.  There  are 
several  possibilities.  The  students  could  be  supplied  with  a  list  of 
programmed  policies  and  they  would  make  their  choices  from  the  list. 

Another  possibility  is  to  let  the  students  devise  their  own  policies 

* 

and  write  subroutines  to  use  these  programs.  Of  course,  the  students 
could  make  their  own  choices  without  the  use  of  programmed  policies. 

Or,  they  could  play  against  a  programmed  policy. 

A  new  scheme  for  contract  negotiation  might  be  established  using 
the  information  generated  by  the  simulation  of  project  histories.  The 
results  obtained  in  the  present  work  yield  a  good  example.  The  results 
shown  in  Table  5- (page  60)  can  be  viewed  as  the  outcomes  under  a  two- 
person,  non-zero-sum  game.  One  player  is  the  customer  whose  choices 
are  the  contracts,  and  the  other  player  is  the  contractor  whose  choices 
are  the  decisionmaking  policies.  The  results  of  contract  negotiation 
would  specify  the  contract  and  the  decisionmaking  policy  to  be  used 
for  the  development. 

A  very  interesting  related  study  would  be  to  have  several  proj¬ 
ect  engineers  select  actions  using  the  hypothetical  project  described 
in  this  study  and  compare  their  results  to  the  results  obtained  here. 

The  value  of  such  a  study  would  be  greatly  enhanced  by  interviewing 
the  engineers  to  determine  how  they  arrived  at  their  decisions. 

The  suggestion  of  using  experienced  engineers  to  study  decision¬ 
making  methods  raises  the  question  of  using  the  methods  developed  in 
this  study  in  real-world  situations.  While,  at  this  time,  it  would  be 
impractical  to  use  any  of  the  decisionmaking  policies  studied  in  this 
research  as  the  basis  for  making  real-world  decisions,  the  information 
structure  might  be  very  useful.  At  present,  sensitivity  analysis  is 
used  to  determine  the  extent  of  variation  in  system  performance  for 
given  variations  in  component  characteristics.  The  information  system 
described  in  this  study  is  a  risk  analysis  system  and  would  enhance 
the  value  of  such  sensitivity  information  because  it  would  show  not 
only  the  amount  of  variation  in  system  performance,  but  it  would  also 
give  information  on  how  likely  such  variations  would  be. 

★ 

As  will  be  shown  in  Appendix  C,  only  five  routine  changes,  at 
most,  are  necessary  to  change  the  policy. 
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Appendix  A 

AIRCRAFT  FUEL  ECONOMY  PERFORMANCE 


This  Appendix  presents  the  derivations  of  the  expressions  for  the 
fuel  economy  performance  of  the  aircraft  for  the  mission  profile  shown 
in  Sec.  I  of  this  study. 

AIRCRAFT  PERFORMANCE* 

Level  flight  of  the  aircraft  at  a  constant  speed  requires  that 
the  total  thrust,  T9  equals  the  total  drag,  Dy 

T  *  Df  (34) 

and  that  the  lift,  Ly  equals  the  instantaneous  gross  weight,  W9 

L  »  W.  (35) 


The  drag,  in  pounds,  can  be  expressed  in  terms  of  the  speed  of  the 
aircraft  in  knots,  Vt  the  wing  area  in  square  feet,  5,  the  altitude 
density  ratio,  0,  and  the  drag  coefficient  as 


D 


c 

294.8  D' 


(36) 


Using  the  parabolic  polar  form  for  the  drag  coefficient. 


CD  ■  cDf  +  7-^7- 


(37) 


the  thrust,  which  equals  the  drag  by  Eq.  (34),  can  be  expressed  as 


* 

The  material  under  this  subheading  is  basic  aerodynamic  theory 
of  sub-sonic,  level  flight.  For  a  more  detailed  exposition  the  reader 
is  referred  to  Courtland  D.  Perkins  and  Robert  E.  Hage,  op.  cit., 
pp.  1-97. 
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T 


05k2 

294.8 


it  fR.  e  I* 


(38) 


where  Cn  =  the  coefficient  of  frictional  drag, 

f 

C ^  =  the  lift  coefficient, 

^  =  the  aspect  ratio  of  the  wing, 
e  *  the  airplane  efficiency  factor, 
it  »  3.14159...  .. 

There  is  an  expression  similar  to  Eq.  (36)  for  the  lift  and  the 
lift  coefficient.  Combining  this  with  Eq.  (35)  gives 


294. 8^ 
05k2  ’ 


(39) 


The  thrust,  in  pounds ,  that  is  required  for  level  flight  at  speed  V 
and  at  an  altitude  corresponding  to  a  is  found  by  substituting  (39) 


into  (38) 


T 


294.8 


+ 


1 

n  R  e 


294.8 W 

aSV2 


(40) 


To  change  pounds  of  thrust  to  horsepower,  the  thrust  is  multiplied  by 
the  speed  in  knots  and  divided  by  325.6.  Hence,  the  thrust  horsepower 
required,  TWP^,,  is  given  by 


THP 


oSk3 


R  325.6  x  294.8 


Cn  +  — ^ — 
D„  tt  e 


294.8V 


35k2 


(41) 
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ENGINE  PERFORMANCE* 

The  expression  for  the  thrust  horsepower  available  per  engine  is 
embodied  in  a  set  of  instructions  for  calculating  the  engine  perfor¬ 
mance  on  pages  38-40  of  the  Allison  Model  Specification  for  the  T56- 
A-10W/10WA  Engine.^  In  addition,  an  equation  for  the  fuel  flow  is  nec¬ 
essary.  Following  is  a  list  of  symbols  that  will  be  used  in  these 
equations : 

n  *  propeller  efficiency  (X). 

P 

Dd  *  duct  efficiency  (X). 

d  ,  ■  engine  outlet  static  pressure  (In.  Hg  Abs). 

S7 

p  •  ambient  pressure  (In.  Hg  Abs). 

am 

p  ■  sea-level  pressure  (In.  Hg  Abs). 

o 

RTF  ■  ram  temperature  ratio. 

RPR  ■  ram  pressure  ratio. 

SRP  *•  shaft  horsepower. 

s  ■  pt 2V 

kSHP  *  change  in  SHP. 

K 2  *  ram  correction  for  shaft  horsepower. 

7^2  *  compressor  inlet  temperature  ( °F ). 

T ^  *  turbine  inlet  temperature  (°F). 

* 

The  material  contained  under  this  subheading  is  taken  from  Model 
Specification  No.  479-D,  Navy  Model  T56-A-10W/10WA ,  Allison  Division 
of  General  Motors  Corp.,  Indianapolis,  Indiana,  2  November  1964.  The 
symbols  are  the  same  as  used  by  Allison. 

+Ibid. 
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T  =  ambient  temperature  (°/?)* 
am 

HPE  =  horsepower  extracted. 

V  =  airspeed  (knots). 

F  =  gross  thrust  (lb) 

0 

K ,  =  ram  correction  for  F  . 

1  9 

tf  =  air  flow  (lb/sec). 
a 

Wj.  =  fuel  flow  (lb /hr). 


The  thrust  horsepower  available  per  engine  is  given  by 


THP, 


WeTPflrt”'* 


3.5 


(42) 


The  quantities  in  parentheses  with  subscripts  that  run  from  14  to  17 

refer  to  engine  performance  items  that  are  presented  as  graphs  in  the 

* 

imdel  specification.  The  numerical  subscripts  correspond  to  the 

number  of  the  figure  in  the  model  specification.  In  order  to  proceed 

with  the  propagation  of  error  technique,  it  is  necessary  to  express 

the  quantities  from  Figs.  14-17  in  that  publication  in  analytical  form. 

Figure  14  shows  F/6K  as  a  function  of  T,0  and  T,c.  It  also  shows 
g  i  tit  5 


* 

Ibid. 
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K .  as  a  function  of  RPR .  Figure  15  shows  W  /&  as  a  function  of  T 
1  a  ti 

and  .  Figure  17  shows  &SHP/K ^6  aS  3  ^unct;*-on  ^ tl*  ^ t5 9 

and  RPR .  It  also  shows  K ^  as  a  function  of  T ^  and  ^^5* 

The  compressor  inlet  temperature,  T can  he  expressed  as 

7\  0  =  T  +  2.369  x  10'  -V  -  459.7,  (43) 

1 1  am 

and  the  ram  temperature  ratio  can  be  expressed  as 

RTR  =  1  +  2.369  *  lCf'V/T’  .  (44) 

am 

Hence,  all  the  quantities  in  Figs.  14-17  can  be  written  as  functions  of 
Vy  and  the  atmospheric  constants.  Furthermore,  6  can  be  written 

as  a  function  of  V  and  the  atmospheric  constants.  Consequently,  Eq .  (42) 
can  be  written  as  a  function  of  V ,  T^,-,  and  the  atmospheric  constants. 

The  equations  for  the  quantities  in  Figs.  14-17  were  obtained  by 
determining  the  range  of  values  of  the  independent  variables  for  which 
a  fit  was  desired,  selecting  a  form  for  the  equation,  reading  a  set  of 
points  off  the  graph,  and  making  a  least-squares  fit.  For  most  of  the 
curves  a  linear  approximation  appeared  reasonable  over  limited  ranges 
of  the  independent  variables.  Because  the  aircraft  performance  is  of 
interest  at  an  airspeed  of  200  knots,  it  is  necessary  to  fit  the  curve 
only  for  a  short  range,  about  >  corresponding  to  V  =  200  knots  and 
T  -  513.4°/?.  The  resultant  equations  are: 

0777 

K  *  .1621  +  .9U5RPR  -  .0763 RPR2.  (45) 

F 

236.24  -  .72 +  .3764 -  .0012 (46) 
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u 

-y  =  36.51  -  .Q101Ttr 


SHP 


-4574. A  +  8.2Tt2  +  5.2387^  -  .0147 


K2  =  1.175  -  .0004Tt2  +  .000117^. 


AS//P 


6*2  /Tt2 


=  107.4  -  104 


(T_tb  +  459 ,7\  -  105.44PPP 
\Tt2  +  459'7' 


/71  +  459. 7\  tT  +  459. 7  \ 

+  103.01PPP  ^  t  459 77 j  +  12 *44  (rt2-+  459.7) 

(T  +  459.7V2 

-  12'15RPR[^n^T7)  • 


(47) 

(48) 

(49) 


(50) 


The  equation  for  the  fuel  flow  is  determined  from  Fig.  18  of  the  model 
specification.  The  result  is 


*JL 

6 


-1011.42  -  2.450 T  +  2.007^  -  .0053 T  T  . 


(51) 


SYSTEM  PERFORMANCE 


The  number  of  nautical  miles  traveled  per  pound  of  fuel  consumed, 
NMI/LB ,  is  found  by  dividing  the  airspeed  (in  knots)  by  the  fuel  flow 
(in  pounds  per  hour).  If  is  the  fuel  flow  for  one  engine,  then  NWj. 
is  the  fuel  flow  for  N  engines.  Equation  (51)  gives  1)^6;  hence,  NMI/LB 
can  be  expressed  as 


NMI 

LB 


N6 


Gf) 


(52) 


and  6  is  given  by 


(53) 


.  wX5 

o  * - '. 

RTR  is  given  by  Eq.  (44)  and  by  Eq.  (51).  Hence,  to  determine 

MI  /LB ,  it  is  necessary  to  know  T  All  other  quantities  are  known 

characteristics  of  the  engines  or  the  flight  conditions. 

* 

To  determine  an  expression  for  the  thrust  horsepower  re¬ 

quired  is  set  equal  to  the  thrust  horsepower  available  from  all  opera¬ 
ting  engines. 


THPr  -  N  *  7WP  .  (54) 

Making  all  the  necessary  substitutions,  measuring  all  temperatures 
in  degrees  Rankine,  and  solving  for  T ^  yields  the  following  cubic 
equation : 

MV®3  +  ^2^t5°^2  +  +  ■  0  (55) 

where 

k3  =  VC3(B6  +  B7BP/?  +  (56) 

k2  =  V  ((B6  +  B7PPP  +  B9RPrZ)  iI<7l/(rt20/?)^] 

0“  )  9  A-  \ 

+  [C2/(7’t2°/?)2  ]j  +  (B2  +  B'RPR  +  B^PR^){C2lTt2°R)2)\,  (57) 


* 

The  turbine  inlet  temperature,  r ,  corresponds  to  the  engine 
power  setting.  As  the  power  level  varies,  T ^  varies.  See  Figs.  34 
and  35  of  the  model  specification,  Ibid. 

+ 

Temperature  in  degrees  Rankine  equals  temperature  in  degrees 
Fahrenheit  plus  459.7. 
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kl  =  V  |(/13  +  \Tt2°R)  + 


1 2 


(#2  +  B^RPR  +  Bq RPR2) 


X  f';2  +  Cl,('Tt2°R'>  J  +  C3(B1  +  B3RPR  +  55/?Pff?)|  |  + 
x  (E1  +  P2/?PP  +  E3RPR2)(D '3  +  ^2°P), 


K6 


325.6 


*0  =  VTl+  ^2?  t2R  +  ^20/?)2(Sl  +  53PP*  +  VP*2> 


and 


X  (C1  +  Vt2^\  + 


K6 


325.6 

V 


( Ei  +  E2 RPR  +  E  RPR2) 


(D1  +  V«?®  -  sf®  <'i  +  V«**|  -  n, 

1  /  294 . 8^  \2 

"  *  6  l  aPK2  ) 


HPE 


aSV 


325.6  x  294.8 


V 


Z? 


5, 


5 


3  . 


5, 


B, 


B . 


-  -4574.4  -  459.7(8.2  +  5.238)  +  459. 72(-.014) 
"  8.2  -  459. 7(-.014)  , 

"  5.238  -  459. 7(-. 014), 

=  -.014, 

■  -35.05, 

!  14.02, 

1  151.33, 

-110.18, 

-116.12, 

-14.77, 

36.81, 

96.08, 

-22.03, 


(58) 


(59) 


(60) 

(61) 

(62) 

(63) 

(64) 

(65) 

(66) 

(67) 

(68) 

(69) 

(70) 

(71) 

(72) 
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C1  =  1.175  -  459. 7(-. 0004  -  .00011),  (73) 

C2  =  -.0004,  (74) 

Cj  =  -.00011,  1  (75) 

Dl  -  236.24  -  459 . 7 (— . 72  +  .3764)  +  459 . 72 (-.0012)  ,  (76) 

D2  -  -.72  -  459. 7(-. 0012)  ,  (77) 

D2  ■=  .3764  -  459. 7(-. 0012)  ,  (78) 

-  -.0012,  (79) 

=  .1621,  (80) 

E2  =  .9145,  (81) 

E3  =  -.0763.  (82) 


The  accuracy  of  these  equations  in  determining  the  various  engine 
performance  items  and  the  fuel  economy  figure  was  determined  by  com¬ 
paring  the  results  of  several  calculations  obtained  U9ing  the  equations 
to  the  results  obtained  using  the  procedure  described  in  the  model 
specification  for  the  same  sets  of  conditions.  In  all  cases,  the  dis¬ 
crepancy  was  less  than  10  percent,  so  the  equations  were  considered  to 
be  sufficiently  accurate  for  the  purposes  of  this  study. 

For  the  computer  program,  a  further  simplification  was  made.  It 
was  observed  in  the  accuracy  checks  that  the  cubic  term  of  Eq.  (55) 
made  a  contribution  of  the  order  of  1  percent  to  the  results.  For 
many  of  the  accuracy  check  cases,  this  was  much  less  than  the  discrep¬ 
ancy;  consequently,  the  cubic  term  was  dropped  in  order  to  simplify 
the  calculations.  The  resulting  equation  is 


k2('Tt5  ^  +  klTt5  R  +  k0  325.6  x  294.8 N 


,  .  1  /294.8y\2 

Df  oSV1) 


0. 


(83) 
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Appendix  B 


DERIVATIONS 


PROPAGATION  OF  UNCERTAINTY  EXPRESSIONS 

This  section  of  Appendix  B  will  present  the  expressions  used  in 
the  propagation  of  uncertainty  calculations.  They  will  be  organized 
according  to  the  hierarchy  of  design  equations  as  shown  in  Fig.  4. 

Airframe  Weight 

The  airframe  weight,  W^9  is  the  sum  of  the  weights  of  the  two  major 

structures  members,  the  wing  and  nacelles,  W  ,  and  the  fuselage  and 

w 


tail,  Wp. 


(84) 


Hence,  the  propagation  of  uncertainty  yields 


«y  ■  +  *<y 


(85) 


and 


o2(Wa)  =  a2(Wu)  +  o2(Wp). 


(86) 


Airframe  Airplane  Efficiency  Factor 


The  airplane  efficiency  factor  at  the  airframe  level,  ,  is  the 
same  as  at  the  wing  level,  e  •  Hence, 


e 


A 


(87) 


The  propagation  of  uncertainty  yields 
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1 

i 


«V  -  *(«„>. 


o2(^)  -  °2<^). 


(88) 

(89) 


Alrfiame  Frictional  Drag  Coefficient 

In  this  research,  it  has  been  assumed  that  the  frictional  drag 
coefficient  of  the  airframe,  ^DjA  9  *S  eclua^  to  sum  t^ie  fr*c“ 

tional  drag  coefficient  of  the  wings,  C  ,  plus  the  frictional  drag 

U  AJ 

coefficient  of  the  fuselage  and  tail,  Cn  In  reality,  a  very  large 

V 

compcnent  of  the  frictional  drag  is  generated  by  interference  between 
the  fuselage  and  the  wings.  As  the  hypothetical  project  was  set  up 
for  this  study,  the  fuselage  included  a  sufficient  stub  of  the  wing 
to  account  for  the  interference  without  including  any  drag  due  to  the 
wing  itself.  Hence,  the  frictional  drag  coefficient  for  the  fuselage 
and  tail  includes  the  interference.  This  simplifies  the  equations 
and  measurement  problem.  Hence, 


CDfA  “  CDjF  +  CDjU 


(90) 


and  the  propagation  of  uncertainty  gives 


Wy)  '  £'czy>  +  £<£y>  • 

o\cD^  -  „2(cy  ♦  »2(cy 


(91)  ’ 

(92) 


Aircraft  Gross  Take-off  Weight 

The  gross  take-off  weight  of  the  aircraft,  GTO  WT ,  is  the  sum  of 
the  weights  of  the  engines,  the  airframe,  W. ,  and  the  other  sub¬ 

systems  that  make  up  the  "flight-ready"  aircraft,  W Hence, 
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GTO  WT=W+W+Wn 
E  A  0 


(93) 


and  the  propagation  of  uncertainty  gives 


E(GT0  WT )  =  E(We)  +  E(Wa)  +  E(WQ ) , 

o2(gto  wt)  =  o2(we)  +  o2(wa)  +  o2(y0). 


(94) 

(95) 


Aircraft  Airplane  Efficiency  Factor 

Because  the  efficiency  factor  is  the  same  at  all  levels,  the  ef¬ 
ficiency  factor  at  the  aircraft,  or  system,  level,  en>  is  the  same  as 
at  the  airframe  level,  eA.  Therefore, 


es  =  V 


(96) 


Hence,  the  propagation  of  uncertainty  yields 


E(es)  =  E(eA) , 
o2(e5)  =  o2(eA). 


(9  7) 
(98) 


Aircraft  Frictional  Drag  Coefficient 

The  frictional  drag  coefficient  at  the  aircraft  level,  Cn  c,  is 
the  same  as  at  the  airframe  level,  Cn  ..  Hence, 

Y 


CDjS  CDjA 


(99) 


This  is  for  the  aircraft  in  a  ncleanu  configuration;  e.g., 
landing  gear  up,  etc.  As  two  of  the  engines  are  shut  down  for  the 
mission  that  is  the  basis  of  all  of  these  calculations,  there  is  an 
increase  in  the  drag  due  to  the  two  feathered  propellers.  A  reason¬ 
able  value  of  the  increase  in  Cn  for  this  type  of  aircraft  is  .003. 

In  this  study  the  increment  is  assumed  to  be  present  in  all  designs 
and  it  is  included  in  the  drag  coefficient  for  the  wings. 


1 
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and  the  propagation  of  uncertainty  gives 


(100) 


(101) 


Aircraft  Fuel  Economy 

The  aircraft  fuel  economy,  NMI/LB ,  is  a  function  of  the  perfor¬ 
mance  of  the  engines  and  the  aerodynamic  characteristics  of  the  air¬ 
craft.  The  latter  are  determined  as  described  above.  The  performance 
of  the  engines  was  discussed  in  Appendix  A. 

Rewriting  Eq.  (83)  in  terms  of  THP ^  and  N  gives 


(102) 


Solving  this  for  yields 


1 


(103) 


T 


1 5  2k 


2k 


2 


2 


Denote  the  value  of  T^°R  obtained  from  Eq.  (103)  with  the  plus  sign 


and  converted  to  degrees  Fahrenheit  by  T^,..  Then  Eq .  (52)  for  NMI/LB 
can  be  written  as 


V/N6 


NMI 


(104) 


LB 


-1011.42  -  2.450Tt2  +  2.00T* 


The  turbine  inlet  temperature  can  never  be  zero  on  the  Rankine 
scale;  hence,  because  the  ratio  of  always  positive,  the  first 

term  of  Eq.  (85)  is  always  negative  and  the  plus  sign  must  be  chosen 
to  obtain  a  positive  T^^°R. 


J 
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where  T ^  is  given  by  Eq.  (43),  and  6  is  given  by  Eq.  (53). 

The  expression  for  77/P  contains  the  aerodynamic  characteristics 

of  the  aircraft:  Cr  c  and  e Q.  These  are  the  only  sources  of  uncer- 

L)  rO  O 

tainty  in  Eq.  (104)  as  the  example  in  this  research  is  set  up.  Given 

normal  probability  distributions  for  C ^  c  and  with  means  c) 

and  E(e^),  and  standard  deviations  and  o(e^),  the  probability 

distribution  for  NMI/LB  is  given  by  E(NMI /LB)  and  o (NMI /LB)  with 

E(NMI/LB)  calculated  using  Eq.  (104)  with  E(Cn  c)  and  E(ec)  used  in 

the  expression  for  77/P ,  and 

H 


a (NMI /LB) 


(NMI  \ 
\  LB  I 


3C 


V 


°2<£y>  + 


) 

91  lb) 

[des  \ 

(105) 


Differentiating  Eq.  (104)  to  determine  the  partial  derivatives  in  Eq. 
(105)  is  far  more  involved  than  using  the  chain  rule  and  starting  with 
Eq.  (52).  Following  the  latter  course  of  action  yields 


(106) 
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3C„  „  '  9 CZZ  ^1924-573  -  .013597^  +  4.436417^  -  .00537^,7^) 
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(4.43641  -  .00537  )  — 
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1 5 


3Cy  5Cy 


~fel  +  ~  4/c2fe0  +  i>k2THFR/N^ 
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2  1_  dTHP 
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3  7tfP 
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v 

1  / 294 ■ I 

tt  AR.  e  l  2  /  i 
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(108) 


aSK 


325.6  x  294.8’ 


(109) 


Combining  Eqs.  (109),  (108),  (107),  and  (106)  yields 


)(W| 

'  LB  I 


3  C 


D 


Y 


N6 


ft) 


(4.4361  - 


•00537t2) 


(*f  - 


4^0  +  Uk2THPR/NS>' 
‘  " 


li 


crsr 


~325.6  X  294.8’  (110) 


Similarly , 


u(i) 


(4.4361  - 


. 005 37^ 2 ) 
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X 


(111) 


In  both  Eqs .  (110)  and  (111),  7 'HP^  is  evaluated  at  E{C^  )  and  E(ec). 


DERIVATION  OF  EXPECTED  PAYOFF  FOR  PERFORMANCE  INCENTIVES 

The  expressions  for  the  expected  payoffs  for  the  two  system  per¬ 
formance  characteristics  that  are  included  in  the  multiple  incentive 
contract  are  quite  similar.  They  differ  only  because  the  direction 
of  the  penalties  and  rewards  are  reversed.  The  basic  expression  for 
both  expected  payoffs  will  be  presented,  but  the  derivation  will  be 
carried  through  for  only  one. 

For  both  performance  characteristics,  the  expected  payoff  has  two 
segments:  the  expectation  over  rewards  for  superior  performance  and 

the  expectation  over  penalties  for  inferior  performance.  Consider  the 
fuel  economy  performance.  Let  x  denote  a  particular  value  of  NMI/LB , 
let  y  denote  E(NMI/LB ),  and  let  o  denote  o(NMI/LB).  Then 


EPANMI  =  RPSNMI  f  — - 

PSRNMT  '  2tt  a 


PSRNMI 


-  CPSNMI  f 


PSRNMI 


( PSRNMI  -  x) 

- - — -  exp 

/ 2tt  a 


2a 


Similarly  for  GTO  WT: 


EPAGTO  =  RPSGTO  f 


PSRGTO 


-  CPSGTO  f  — - PSRGTQ1 

PSRGTn  Su  a 


PSRGTO 


oo 


dx,  (113) 
2  a 
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where  x,  p,  and  a  in  Eq.  (113)  refer  to  GTO  WT. 
Proceeding  with  Eq.  (112),  let 

y  m  (x  -  p)/a. 


Then 


x  =  ay  +  p, 
dx  »  a  dy , 


and  when  x  =  PS UN MI , 


y  =  (PSRNMI  -  p)/a. 


Making  these  substitutions  into  Eq.  (112)  yields 


EPANMI  =  PPSAWI  f 


(ay  +  p  -  PSRNMI} 


-  CPSNMI  f 


( PSRNMI-v ) /a 
(PSRNMI-v) /a 


/2iT  a 
jPSRNMI  -ay  -  p) 


exp 


(-4) 


exp 


(-4 


Expanding  the  parentheses  within  the  integrals  and  collecting 
terms  yields 


EPANMI 


RPSNMI 


r—  f  y  exp 

/2tt  (PSRNMI-v) /a 


U) 


qCPSNMI 

/2v 


f 


(PSRNMI-v) /a 


y  exp 


dy 

(-4) 


dy 


RPSNMI (y  -  PSRNMI)  f  /  y^_  \ 

k —  *!  exP  V-  9  / 

/2lf  (PSRNMI-v)  /a  A 

_  CPSim(PSRNMI  -  u)  SPSRNMI~v)  h  /  I  2  \ 

^7  J_»  exp  2  ' 


( HA) 

(115) 

(116) 

’  dy 

)  a  dy. 

(117) 

some 

dy 

dy.  (118) 
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It  should  be  recognized  that 


exp  (-#  /2) 

JlS 


f(y) 


(119) 


is  the  normal  density  function  with  zero  mean  and  unit  variance.  Also, 

2  2 
y  exp  (-y  /2)  dy  is  the  exact  differential  of  -exp  (-y  /2).  Hence, 

Eq.  (118)  can  be  rewritten  as 


EPANMI 


oRPSNMI 

Sir, 


exp 


(-v) 


/  „2\ 

(PSRNMI-v) /o' 

-exp  (-  Uj) 

— oo 

qCPSNMI 

(PSRNMI-m) /a J  ^ 

+  RPSNMIiv  ~  PSRNMI ) 


oo  z 

f  e.xE.i.~lL-/.2I  dy  _  CPS  mi  (PSRNMI  -  v) 

(PSRNMI-v) /a  *2* 


1  “  J  exp  (-  ^-)  dy 


(PSRNMI-v) /a 

Now  it  should  be  noticed  that 


(120) 


-jz  f  exp  (-v2)  dv  -  ERFC(z) 
/tt  z 


(121) 


is  the  complementary  error  function  of  z.  Hence,  let 


y  2 
2  =  V 


(122) 


Then 


y  =  Sl  v. 


(123) 
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dy  -  dv  (124) 

and  when  y  =  ( PSRNMI  -  p)/cr,  V  ■  ( PSRNMI  -  p)//2  a.  Also  note  that  the 
lower  limit  of  the  second  integral  has  no  meaning  in  reality;  hence,  it 
is  replaced  by  zero.  Now  Eq.  (120)  can  be  rewritten  as 


( PSRNMI-U )2 

a CPSNMI 

(PSfliWfl-u) 2 

2 

a  J 

/r —  exp 

/2tt 

2 

a  J 

oRPSNMI 

EPANMI  =  — —  e: 

y2v 

.  oCPSNMI  .  ( RPSNMI  -  CPSNMI) (u  -  PSRNMI)  „„„„  l PSRNMI 

+ ~~7r~  2 


The  expression  for  EPAGTO  is  derived  similarly  beginning  with 
Eq.  (113).  The  result  is 


EPACTO  ,  '(CPSOTO-  RPSGTO I  exp 

/2tt 


(PSRGTO-u)' 

2 

a 


-  RPSGTO  /  V  -  PSRGTO  +  -7= 


y 


/2r 

( CPSGTO  -  RPSGTO) {PSRGTO  -  p) 

2 


ERFC 


I  PSRGTO  - 

l  ^ 


*)• 


(126) 


DERIVATION  OF  THE  PROBABILITY  OF  TECHNICAL  SUCCESS 

Technical  success  is  defined  as  both  system  performance  factors 
being  equal  or  superior  to  the  zero  performance  incentive  payoff  levels. 
Specifically,  technical  success  occurs  when 


mi 

LB 


>  PSRNMI, 


(127) 


and 
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GTO  WT  <  PSRGTO . 


(128) 


Because  these  two  technical  performance  factors  are  independent,  the 
probability  that  both  Eq .  (127)  and  Eq.  (128)  will  be  jointly  satis¬ 
fied  is  given  by  the  product  of  the  probabilities  that  they  will  be 
independently  satisfied.  Hence, 

Pr  ( ^  S  PSRNMI J  GTO  WT  £  PSRGTO )  =  Pr  ( ^  ^  PSRNMI ) 

x  Pr(GTO  WT  £  PSRGTO).  (129) 

Probability  That  Fuel  Economy  Exceeds  PSRNMI 

Let 


x  =  NMI/LB, 

(130) 

p  =  E(NMI /LB) , 

(131) 

o  =  o (NMI/LB). 

(132) 

Then 


Pr  (^||  ^  PSRNMl)  = 


•1 


/—  /  exp 

/2’x  o  PSRNMI 


Qc-u)‘ 

'  2C2 


dx. 


(133) 


Making  the  substitutions  shown  in  Eqs.  (114),  (115),  and  (116)  gives 


Pr  (~  *  PSRNMI )  =  -L  f  exp  (-  ^-)  dy. 

LU  /2ir  (PSRNMI- p)/o  1  ' 


(134) 


Again,  recognizing  Eq.  (121)  and  making  the  substitutions  shown  in 
Eqs.  (122),  (123),  and  (124)  yields 


Pr  (^r|  :>  PSRNMl)  =  y  ERFC  |  - Jlj. 


(135) 


-115- 


Probability  That  Gross  Take-off  Weight  Is  Less  Than  PSRGTO 

Proceeding  similarly  to  the  above. 


PSRGTO 

Pr ( GTO  WT  z  PSRGTO )  =  — f  exp 

/2tT  a  oo 


(*-u)‘ 

2  a2 


dx 


becomes 


Pr  ( GTO  WT  <:  PSRGTO) 


1  -  j  ERFC 


PSRGTO  -  w\ 

/2  a  j- 


(136) 
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Appendix  C 

DESCRIPTION  OF  THE  COMPUTER  PROGRAM 

The  computer  program  used  in  this  study  was  written  in  FORTRAN 
IV.  All  runs  were  made  on  an  IBM  360/40  at  The  Rand  Corporation. 
Reproduction  of  the  program  listings  is  not  feasible  as  this  would 
require  approximately  70  pages.  Instead,  the  functions  the  sub¬ 
routines  perform  will  be  discussed  briefly.  Figure  14  shows  the 


MAIN 


EQN303 

Fig .  14--Calling  hierarchy 

relationships  of  the  subroutines  through  the  calling  statements.  To 
change  the  decisionmaking  policy,  only  the  MAIN,  VALUE,  VEXPMT,  VASSEM, 
and  VDELIV  subroutines  need  to  be  changed.  Some  changes  do  not  require 
all  these  subroutines  to  be  changed. 
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MAIN  PROGRAM 

The  main  program  performs  a  large  portion  of  the  bookkeeping 
tasks  including:  (1)  initializing  the  time,  cost,  sequence  numbers, 
probability  distributions,  and  prerequisite  indexes  for  each  project 
simulation;  (2)  writing  the  system  performance  probability  distribu¬ 
tions  prior  to  action  evaluation  and  selection;  (3)  testing  to  see 
that  any  required  actions  have  been  selected;  (A)  calling  subroutine 
VALUE  to  perform  the  evaluation  of  actions  and  select  the  "best"; 

(5)  calling  subroutine  EXINFO,  or  ATINFO,  which  determines  the  new 
states  of  knowledge  for  the  characteristics  measured  by  the  action 

selected;  (6)  recording  the  satisfaction  of  prerequisites;  (7)  de- 

* 

leting  subordinate  actions  from  those  on  the  eligible  list;  (8)  up¬ 
dating  the  present  time  and  cost;  (9)  recording  the  order  of  selection 
for  the  action  just  selected;  (10)  removing  the  action  selected  from 
the  list  of  eligible  actions;  (11)  testing  for  selection  of  a  terminal 
action;  (12)  if  action  selected  is  not  terminal,  returning  to  select 
another  action;  (13)  if  action  is  terminal,  identifying  the  system 
design  chosen  and  calling  OUTPUT  to  record  the  final  state;  and  (1A) 
returning  to  make  another  project  simulation  if  the  sample  is  not 
complete.  The  main  program  also  writes  messages  for  certain  errors 
that  may  occur.  These  errors  will  not  occur  when  the  program  is 
functioning  properly . 

* 

This  is  done  when  an  assemble  and  test  is  selected.  In  this 
case,  perfect  knowledge  is  obtained  regarding  all  components  of  the 
system;  hence,  any  action  that  would  produce  imprecise  information 
regarding  the  components  of  the  design  that  was  assembled  is  unneces¬ 
sary  . 
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SUBROUTINE  "INPUTS11 

This  subroutine  reads  the  input  data  including  the  initial  prob¬ 
ability  distributions  for  the  characteristics  of  the  fuselage  and 
tail  and  the  wing  and  nacelles. 

SUBROUTINE  1  ’INITSP" 

This  subroutine  determines  the  initial  probability  distributions 
for  all  characteristics  of  the  airframe  and  the  aircraft.  It  calls 
subroutine  EQN303,  EQN302,  EQN301,  EQN104,  EQN103,  EQN102 ,  and  EQN101. 

SUBROUTINE  "VALUE" 

This  subroutine  performs  a  number  of  bookkeeping  functions  and 
calls  subroutines  to  evaluate  actions.  The  subroutine  consists  of  a 
"Do  Loop"  ranging  over  all  the  actions.  The  flow  of  the  subroutine 
is  as  follows:  (1)  The  subroutine  is  entered  with  a  low  initial  value 
for  the  maximization  process.  (2)  An  action  is  selected  by  the  Do  Loop. 
(3)  If  the  action  has  been  selected  previously,  it  cannot  be  selected 
again  and  control  returns  to  the  start  of  the  Do  Loop;  however,  if  the 
action  has  not  been  selected  previously,  control  proceeds  to  the  next 
step.  (4)  The  activity  category  of  the  action  is  determined.  If  the 
action  is  from  the  "Deliver"  category,  control  passes  to  Step  5a.  If 
it  is  from  the  "Assemble  and  Test"  category,  control  passes  to  Step  5b. 
Otherwise,  control  passes  to  Step  6c.  (5a)  If  the  prerequisite  assemble 

and  test  action  has  not  been  previously  selected,  control  returns  to  the 
start  of  the  Do  Loop.  If  the  prerequisite  has  been  selected,  the  con¬ 
trol  proceeds  to  Step  6a.  (5b)  If  the  prerequisite  experiment  actions 

have  not  been  previously  selected,  control  returns  to  the  start  of  the 
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Do  Loop;  otherwise,  control  proceeds  to  Step  6b.  (6a)  Subroutine 

VDELIV  is  called  to  determine  the  value  of  the  action  and  control 
proceeds  to  Step  7.  (6b)  Subroutine  VASSEM  is  called  to  determine 

the  value  of  the  action  and  control  proceeds  to  Step  7.  (6c)  Sub¬ 

routine  VEXPMT  is  called  to  determine  the  value  of  the  action  and 
control  proceeds  to  Step  7.  (7)  The  value  determined  at  Step  6  is 

compared  to  the  previous  high  value,  or  the  low  initial  value  the 
first  time  through.  If  the  value  of  the  action  being  evaluated  is 
greater  than  or  equal  to  the  previous  value,  then  the  present  value 
is  substituted  and  the  action  being  evaluated  is  identified  as  the 
action  to  be  selected.  Control  proceeds  to  the  end  of  the  Do  Loop. 
(8)  If  all  actions  have  not  been  evaluated,  control  returns  to  the 
beginning  of  the  Do  Loop;  otherwise,  control  returns  to  the  main 
program. 

SUBROUTINE  "EXINFO" 

This  subroutine  identifies  characteristics  measured  by  an  ex¬ 
periment  action  and  samples  the  corresponding  probability  distribu¬ 
tions  to  determine  the  new  state  of  knowledge  regarding  the  measured 
characteristics.  Subroutine  GAUSS  is  called. 

SUBROUTINE  MATINFOM 

This  subroutine  performs  the  same  functions  as  EXINFO,  but  only 
for  the  characteristics  measured  by  an  assemble  and  test  action. 

* 

Note  that  Step  5b  is  not  present  in  Policy  2. 
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SUBROUTINE  "UPDATE" 

This  subroutine  determines  the  probability  distributions  for 
characteristics  at  the  airframe  and  aircraft  levels  from  lower  level 
distributions.  It  is  used  for  both  updating  after  an  action  has  been 
selected  and  for  determining  preposterior  distributions.  It  calls 
subroutine  EQN303,  EQN302,  EQN301,  EQN104,  EQN103,  EQN102 ,  and  EQN101 
in  the  proper  order. 

SUBROUTINE  ,,VDELIVM 

This  subroutine  determines  values  for  deliver  actions  as  described 
in  Sec.  IV. 

SUBROUTINE  "VASSEM" 

This  subroutine  determines  values  for  assemble  and  test  actions  as 
described  in  Sec.  IV. 

SUBROUTINE  "VEXPMT" 

This  subroutine  determines  values  for  experiment  actions  as  des¬ 
cribed  in  Sec.  IV. 

SUBROUTINE  ,,EQN3Q3,t 

This  subroutine  determines  the  mean  and  standard  deviation  for 
the  airplane  efficiency  factor  at  the  airframe  level.  It  uses  Eqs . 

(88)  and  (89). 

SUBROUTINE  "EQN3Q2" 

This  subroutine  determines  the  mean  and  standard  deviation  for 
the  frictional  drag  coefficient  at  the  airframe  level.  It  uses  Eqs. 

(91)  and  (92). 
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SUBRQUTINE  ,,EQN301,t 

This  subroutine  determines  the  mean  and  standard  deviation  for 
the  weight  of  the  airframe.  It  uses  Eqs.  (85)  and  (86). 

SUBROUTINE  "EQN104" 

This  subroutine  determines  the  mean  and  standard  deviation  for 
the  airplane  efficiency  factor  at  the  system . level .  It  uses  Eqs. 
(97)  and  (98). 

SUBROUTINE  nEQN103" 

This  subroutine  determines  the  mean  and  standard  deviation  for 
the  frictional  drag  coefficient  at  the  system  level.  It  uses  Eqs. 
(100)  and  (101). 

SUBROUTINE  "EQN102" 

This  subroutine  determines  the  mean  and  standard  deviation  for 
the  gross  take-off  weight  of  the  aircraft.  It  uses  Eqs.  (94)  and 
(95). 

SUBROUTINE  ,,EQN101,t 

This  subroutine  determines  the  mean  and  standard  deviation  for 
nautical  miles  per  pound  for  the  aircraft.  It  uses  Eqs.  (104)  and 
(105). 

SUBROUTINE  "GAUSS" 


This  subroutine  determines  random  numbers  drawn  from  specified 
normal  distributions.  It  is  part  of  the  IBM  Scientific  Subroutine 
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Package.  It  calls  subroutine  RANDU. 

SUBROUTINE  M RANDU" 


This  subroutine  determines  uniform  random  numbers.  It  also  is 
part  of  the  Scientific  Subroutine  Package. 


SUBROUTINE  "OUTPUT" 


This  subroutine  writes  the  final  states  and  sequences  of  actions 
of  the  simulation  projects. 


System/360  Scientific  Subroutine  Package  (360A-CM-03X)  Version 
III  Programmer's  Manual 3  4th  ed.  ,  H20-0205-3,  International  Business 
Machines  Corporation,  New  York,  1968. 

+Ibid. 
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Appendix  D 

MONTE  CARLO  VS  PROPAGATION  OF  ERROR 

i  h  '  5  ' 

There  are  two  factors  involved  in  the  choice  between  using  Monte 

Carlo  methods  (MC)  and  using  the  Propagation  of  Error  method  (PE)  to 
determine  aggregate  uncertainty  of  error.  These  factors  are  the  close¬ 
ness  between  the  results  obtained  using  the  two  methods  and  the  amount 
of  computer  time  available.  The  reason  that  there  may  be  differences 
between  the  results  obtained  with  the  two  methods  is  that  the  probabil¬ 
ity  distributions  involved  may  not  be  normal.  MC  can  be  used  with 
probability  distributions  having  any  form  while  PE  requires  that  all 
distributions  be  normal;  hence,  in  situations  involving  nonnormal 
distributions,  MC  calculations  with  large  samples  should  yield  results 
that  are  closer  to  "reality. " 

There  are  several  factors  that  influence  the  normality  of  the 
probability  distributions  involved  in  any  given  study.  One  factor  is 
the  shape  of  the  low-level  distributions.  In  the  present  study,  they 
are  assumed  to  be  normal.  Three  other  factors  relate  to  the  shape  of 
the  aggregate  distributions.  These  are  the  equations  that  specify  the 
relationship (s)  between  the  low-level  variables  and  the  high-level 
variables,  the  central  values  of  the  low-level  distributions,  and  the 
spreads  of  the  low-level  distributions. 

In  the  present  study,  all  equations  involved  in  the  propagation 
of  uncertainty  calculations  are  linear  except  for  the  fuel  economy 
equation.  No  significant  difference  between  MC  and  PE  is  expected 
for  linear  equations  because  the  partial  derivatives  of  the  high- 
level  variables  with  respect  to  the  low-level  variables  are  all  con¬ 
stants.  Consequently,  the  evaluation  will  not  depend  on  any  mean 
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values  for  low-level  variables.  To  compare  MC  to  PE  for  the  fuel 
economy  equation,  Eq.  (104),  samples  of  1000  calculations  were  made  and 
the  mean  and  standard  deviation  of  the  results  were  compared  to  the 
mean  and  standard  deviation  obtained  using  PE.  Three  different  low- 
level  uncertainties  were  used.  The  results  are  summarized  in  Tables 
13  and  14.  The  gross  take-bff  weight  was  also  included  as  a  check  on 
the  sampling  process. 

The  results  are,  of  course,  dependent  on  the  random  number  gener¬ 
ator  used.  In  the  present  study x  the  random  number  generator  used  is 

that  contained  in  the  IBM  Scientific  Subroutine  package  for  the  360 
* 

series  computers.  Given  this  means  of  generating  normal  random  num¬ 
bers,  it  was  judged  that  Case  I  and  Case  II  gave  acceptable  results 
and  Case  III  did  not. 

The  initial  uncertainties  for  A  and  e A  that  were  used  in  this 

DA  A 

study  are  summarized  in  Fig.  7.  All  values  are  reasonably  close  to 
or  less  than  the  values 

*Ibid. 


for  Case  II. 
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Table  13 

LOW-LEVEL  PROBABILITY  DISTRIBUTIONS  USED  TO  COMPARE 


MONTE  CARLO  AND  PROPAGATION  OF  ERROR 


Variables 

Case  I 

Case  II 

Case  III 

.024 

.024 

.024 

0<cc/s) 

.0024 

.0048 

.0072 

E(es) 

.9 

.9 

.9 

a,eS) 

.09 

.18 

.27 

E(V 

27000. 

27000. 

27000. 

cWA) 

2700. 

5400. 

8100 : 

Table  14 


RESULTS  OF  MONTE  CARLO  AND  PROPAGATION  OF  ERROR  CALCULATIONS 
FOR  LOW-LEVEL  UNCERTAINTIES 


Variables 

MC 

PE 

MC-PE 

(MC-PE) 
x  100 /MC 

Case  I 

E (NMI/LB) 

.080908 

.080982 

-.000074 

-.09 

a(NMI/LB ) 

.004428 

.004240 

.000188 

4.24 

E(GTO  WT) 

126888. 

127000. 

-112. 

-.09 

a(GTO  WT) 

3019. 

2700. 

319. 

10.57 

Case  II 

E (NMI/LB) 

.080815 

.080982 

-.000167 

-.21 

a  (NMI/LB) 

.009110 

.008481 

.000629 

6.90 

E(GTO  WT) 

126784. 

127000. 

-216. 

-.17 

a(GTO  WT) 

569  7. 

5400. 

29  7. 

5.21 

Case  III 

E (NMI/LB) 

.080612 

.080982 

-.000370 

-.46 

a (NMI/LB) 

.014798 

.012721 

.002077 

14.04 

E(GTO  WT) 

126679. 

127000. 

-321. 

-.25 

a(GTO  WT) 

8453. 

8100. 
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